AFRL-HE-AZ-TR-1 998-01 1 1 


UNITED  STATES  AIR  FORCE 
RESEARCH  LABORATORY 


AN  INTEGRATED  PRODUCT  AND  PROCESS  DEVELOPMENT 
PROGRAM  AND  RELATED  TRAINING  REQUIREMENTS: 
FORMATIVE  EVALUATION  AND  DEFINITION  FOR  AIR  FORCE 
LABORATORY  SCIENCE  AND  ENGINEERING  PERSONNEL 


Jerry  Barucky 
Gary  Grimes 
Donna  Tartell 


Metrics,  Inc. 

10010  San  Pedro  Avenue,  Suite  400 
San  Antonio  TX  78216-3856 


Robert  M.  Yadrick 


Air  Force  Research  Laboratory 
Warfighter  Training  Research  Division 
7909  Lindbergh  Drive 
Brooks  Air  Force  Base  TX  78235-5352 


November  1999 

20000711  118 


Approved  for  public  release;  distribution  is  unlimited. 


ETIC  QUALITY  INSPECTED  4 


AIR  FORCE  MATERIEL  COMMAND 
AIR  FORCE  RESEARCH  LABORATORY 
HUMAN  EFFECTIVENESS  DIRECTORATE 
WARFIGHTER  TRAINING  RESEARCH  DIVISION 
6030  South  Kent  Street 
Mesa  AZ  85212-6061 


NOTICES 


Using  Government  drawings,  specifications,  or  other  data  included  in 
this  document  for  any  purpose  other  than  Government-related  procurement 
does  not  in  any  way  obligate  the  US  Government.  The  fact  that  the 
Government  formulated  or  supplied  the  drawings,  specifications,  or  other 
data,  does  not  license  the  holder  or  any  other  person  or  corporation,  or 
convey  any  rights  or  permission  to  manufacture,  use,  or  sell  any  patented 
invention  that  may  relate  to  them. 

The  Office  of  Public  Affairs  has  reviewed  this  report,  and  it  is  releasable 
to  the  National  Technical  Information  Service,  where  it  will  be  available  to  the 
general  public,  including  foreign  nationals. 

This  report  has  been  reviewed  and  is  approved  for  publication. 


ROBERT  M.  YADRICK 
Contract  Monitor 


DEE  H.  ANDREWS 
Technical  Director 


JERALD  L.  STRAW,  Colonel,  USAF 
Chief,  Warfighter  Training  Research  Division 


REPORT  DOCUMENTATION  PAGE 

Form  Approved 

OMB  No.  0704-0188 

Public  reporting  burden  for  this  collection  of  information  is  estimated  'o  average  1  regafdfng°lhis' burden  estimate  or'any  other9 aspect  of  this  collection  of  | 

and  maintaining  the  data  needed.  D^SS^f^Infor^ation  Operations  and  Reports,  1215  Jefferson  Dav.s  Hrghway.  Surte 

Pro<ect  <0704-°188>'  Washington,  DC  20503.  . . J 

1.  AGENCY  USE  ONLY  (Leave  blank)  2.  REPORT  DATE  3-  R^P°RT  T^P^N' 

November  1 999  Final  -  May  1 997  to 

D  DATES  COVERED  j 

May  1998  ! 

4.  TITLE  AND  SUBTITLE 

An  Integrated  Product  and  Process  Development  Program 
and  Related  Training  Requirements:  Formative  Evaluation 
and  Definition  for  Air  Force  Laboratory  Science  and  Engineering  Personnel 

5.  FUNDING  NUMBERS  ! 

} 

C  -  F41624-97-C-5012  ! 

PE  -  62205F 

PR  1  1 0-I 

6.  AUTHOR(S) 

Jerry  Barucky,  Gary  Grimes,  Donna  Tartell,  &  Robert  M.  Yadrick 

r  Iv  I 

TA  -  A3 

WU  -  23 

7.  PERFORMING  ORGANIZATION  NAME(S)  AND  ADDRESS(ES) 

Metrica,  Inc. 

10010  San  Pedro  Avenue,  Suite  400 

San  Antonio  TX  78216-3856 

8.  PERFORMING  ORGANIZATION 
REPORT  NUMBER 

9.  SPONSORING/MONITORING  AGENCY  NAME(S)  AND  ADDRESS(ES) 

Air  Force  Research  Laboratory 

Human  Effectiveness  Directorate 

Warfighter  Training  Research  Division 

6030  South  Kent  Street,  Bldg  561 

Mesa  AZ  85212-6061 

10.  SPONSORING/MONITORING 

AGENCY  REPORT  NUMBER  j 

AFRL-HE-AZ-TR-1998-01 1 1 

11.  SUPPLEMENTARY  NOTES 

Air  Force  Research  Laboratory  Contract  Monitor:  Dr  Robert  M.  Yadrick, (210)  340-8211;  DSN  487-681 1  X-3.035 

12a.  DISTRIBUTION/AVAILABILITY  STATEMENT 

Approved  for  public  release;  distribution  is  unlimited. 

12b.  DISTRIBUTION  CODE 

S 

13.  ABSTRACT  (Maximum  200  words)  j 

In  recent  years,  the  United  States  Department  of  Defense  and,  by  extension,  the  Air  Force  have  increased  their  emphasis  on 
considerations  of  affordability  in  the  weapons  development  cycle,  particularly  in  the  Science  and  Technology  (S&T)  phase.  To 
help  its  members  attain  a  better  balance  between  performance,  cost,  and  risk  in  their  research  and  development  efforts,  the  Air 
Force  has  encouraged  the  application  of  Integrated  Process  and  Product  Development  (IPPD)  principles.  The  approach 
involved  selecting  a  group  of  pilot  projects  at  several  locations  in  the  Air  Force  Research  Laboratory  (AFRL)  system  and 
providing  a  training  program,  Affordable  Technology  Through  IPPD  (AT/IPPD),  to  key  members  of  those  pilot  projects.  The 
Air  Force  Materiel  Command  (AFMC)  also  funded  the  present  effort  to  provide  a  formative  evaluation  of  that  IPPD  training 
program.  The  objectives  of  this  research  were  to  gather  and  analyze  data  related  to  the  effectiveness  of  the  AT/IPPD  framing 
program,  determine  the  extent  to  which  the  training  participants  were  able  to  implement  the  IPPD  procedures  in  their  projects, 
and  identify  obstacles  to  implementation  of  IPPD  in  S&T  programs. 


14.  SUBJECT  TERMS  ,  _  .  . 

Air  Force  Research  Laboratory;  Integrated  Product  and  Process  Development;  IPPD;  Training; 

Training  Effectiveness;  Training  Evaluation; 

15.  NUMBER  OF  PAGES 

109 

16.  PRICE  CODE 

17.  SECURITY  CLASSIFICATION 
OF  REPORT 

UNCLASSIFIED 

18.  SECURITY  CLASSIFICATION 
OF  THIS  PAGE 

UNCLASSIFIED 

19.  SECURITY  CLASSIFICATION 
OF  ABSTRACT 

UNCLASSIFIED 

20.  LIMITATION  ABSTRACT 

UL 

NSN  7540-01-280-5500 


Standard  Form  298  (Rev  2-89) 
Prescribed  by  ANSI  Std  Z-39-18 


CONTENTS 


INTRODUCTION .  1 

Background .  1 

Purpose  of  die  Research .  3 

METHODOLOGY .  4 

The  Research  Plan .  4 

Analysis  of  the  AT/IPPD  Training  Sessions .  5 

Analysis  of  the  Implementation  of  IPPD  Procedures  in  the  IPT  Pilots .  7 

RESULTS .  9 

AT/IPPD  Training  Sessions . . .  9 

Implementation  of  IPPD  in  IPT  Pilots .  19 

Participation  in  Supplemental  Training  and  Mentoring  Opportunities .  28 

Perceptions  of  Management’s  Knowledge/Understanding/Attitude/Support .  31 

Perceptions  of  Benefits  to  Projects .  32 

Obstacles  to  Implementation  of  IPPD . 35 

CONCLUSIONS  AND  RECOMMENDATIONS .  36 

The  Initial  AT/IPPD  Training  Session .  38 

Implementation  of  IPPD  in  the  Pilot  Programs .  39 

The  IPPD  Training  Evaluation  Effort .  42 

General  Recommendations .  43 

REFERENCES .  45 

APPENDICES: 

A  Participant  Questionnaire .  47 

B  Interview  Session  Discussion  Questions .  53 

C  Follow-Up  Questionnaire .  55 

D  Summary  of  Processes  Used  Survey .  65 

E  Tables  of  Questionnaire  Responses  by  Pilot  Group .  69 

F  Summary  of  IPPD  Application  Experiences  of  Pilot  IPTs .  85 

FIGURES 

Figure  1 .  Capability  of  IPPD  Application  after  Training .  11 

Figure  2.  IPPD  Additional  Training  or  Mentoring  Needed .  11 

Figure  3.  Prior  Projections  about  Successful  IPPD  Application .  12 

Figure  4.  Post-training  Projections  about  Successful  IPPD  Application .  12 

Figure  5.  Prior  Projections  about  IPPD  Achieving  Real  Benefits .  13 

Figure  6.  Post-training  Projections  about  IPPD  Achieving  Real  Benefits .  13 

Figure  7.  IPPD  Benefits  to  Other  Projects . . .  14 

Figure  8.  Usefulness  of  IPPD  To  Personal/Professional  Development .  14 

Figure  9.  Perception  of  Management’s  Knowledge  of  IPPD  Process .  15 

Figure  10.  Perception  of  Management’s  Understanding  of  IPPD  Initiative .  15 

Figure  1 1 .  Perception  of  Management’s  Understanding  of  IPPD  Benefits .  15 

Figure  12.  Perception  of  Management’s  Understanding  of  IPPD  Application .  16 

Figure  13.  Perception  of  Management’s  Projections  about  Successful  IPPD  Application .  16 

Figure  14.  Perception  of  Management’s  Projections  about  IPPD  Achieving  Real  Benefits .  16 


iii 


TABLES 


Page 

Table  1  DSSM/IPPD  Training  Session  Participants  by  Pilot  Project .  5 

Table  2  Number  Responding  to  IPPD  Training  Session  Evaluation 

Questionnaire  by  Pilot  Project .  5 

Table  3  IPPD  Training  Session  Questionnaire  Respondents  by  Organization .  6 

Table  4  IPPD  Training  Session  Questionnaire  Respondents  by  Position .  6 

Table  5  Members  Participating  in  IPPD  Follow-up  Data  Gathering  by  IPT .  8 

Table  6  Percentage  of  Respondents  Reporting  Various  Amounts  of  IPPD 

Experience  and  Knowledge .  9 

Table  7  Percentage  of  Respondents  Reporting  Amount  of  IPPD  Understanding 

Before  and  After  IPPD  Training .  10 

Table  8  Percent,  By  Pilot,  Indicating  Various  Factors  Hindered  Their  IPPD 

Implementation  a  Large  or  Very  Large  Amount .  17 

Table  9  Percentage  of  Respondents  Reporting  Ability  to  Apply  IPPD  Just  After 

and  Months  After  Training .  20 

Table  1 0  Percentage  of  Respondents  Reporting  Ability  to  Apply  IPPD  Just  After 

and  Months  After  Training .  21 

Table  1 1  Percentage  of  Respondents,  By  Individual  Pilot,  Participating  at  Various 

Levels  in  the  Application  of  IPPD  to  Their  Project .  22 

Table  12  Percentage  of  Respondents,  By  Individual  Pilot,  Participating  at  Various 

Levels  in  Completing  a  House  Of  Quality .  23 

Table  13  Percentage  of  Respondents,  By  Individual  Pilot,  Participating  at  Various 

Levels  in  Use  of  Technology  Worksheets .  23 

Table  14  Percentage  of  Respondents,  By  Individual  Pilot,  Participating  at  Various 

Levels  in  Use  of  Value  Scorecards .  24 

Table  15  Number  of  Pilots  Reporting  Various  Types  of  Customer  Interaction,  from 

Choices  on  Summary  of  Processes  Survey .  24 

Table  16  Number  of  Pilots  Reporting  Various  Means  of  Determining  Requirements 

and  Exit  Criteria,  from  Choices  on  Summary  of  Processes  Survey .  25 

Table  1 7  Number  of  Pilots  Reporting  Extent  of  Use  of  House  of  Quality,  from 

Choices  on  Summary  of  Processes  Survey .  25 

Table  18  Number  of  Pilots  Reporting  Various  Means  for  Comparing  and  Documenting  Design/ 

Technology  Alternatives,  from  Choices  on  Summary  of  Processes  Survey .  26 

Table  19  Number  of  Pilots  Reporting  Various  Means  for  Carrying  Out/Documenting 

a  Value  Analysis  from  Choices  on  Summary  of  Processes  Survey .  27 

Table  20  Number  of  Pilots  Reporting  Extent  of  Development  of  a  Technology  Transition 

Business  Plan,  from  Choices  on  Summary  of  Processes  Survey .  27 

Table  21  Respondents’  Participation  in  IPPD  Supplemental  Courses .  28 

Table  22  Number  of  Mentoring  Incidents  Reported  by  Respondents .  29 

Table  23  Amount  of  IPPD  Knowledge  Reported  by  Respondents  with  Various 

Amounts  of  Mentoring  Communication .  29 

Table  24  Ability  to  Apply  IPPD,  Reported  by  Respondents  with  Various  Amounts 

of  Mentoring  Communication .  29 

Table  25  Extent  of  Participation  in  Application  of  IPPD  to  Project,  Reported  by 

Respondents  with  Various  Amounts  of  Mentoring  Communication .  30 

Table  26  Likelihood  of  Applying  IPPD  In  Future  Projects,  Reported  by  Respondents 

with  Various  Amounts  of  Mentoring  Communication .  30 

Table  27  Percentage  of  Respondents  Reporting  Management’s  Levels  of  IPPD 

Knowledge  and  Understanding .  31 


TABLES  (Cont’d) 


Table  28  Percentage  Of  Respondents  Reporting  Management’s  Perceptions  About  Success 

of  IPPD  Application  in  Pilot  Projects  and  die  Extent  of  Benefits  Gained . 

Table  29  Percentage  of  Respondents  Reporting  Extent  of  Management  Support  for 

IPPD  Training  and  Use  of  IPPD  Methods . .  . . 

Table  30  Respondents’  Perceptions  of  Extent  IPPD  Application  Provided 

or  Would  Provide  Overall  Benefits  to  Their  Project  or  Other  Projects . 

Table  31  Respondents’  Perceptions  of  Extent  IPPD  Application  Benefited 

Specific  Aspects  of  Their  Project . 

Table  32  Respondents’  Perceptions  of  Training  Utility  . . . . 

Table  33  Percent,  By  Pilot,  Indicating  Various  Factors  Hindered  Their  IPPD 

Implementation  a  Large  or  Very  Large  Amount . 


PAGE 

31 

32 

33 

33 

34 

37 


PREFACE 


This  report  documents  a  contract  effort  performed  under  Work  Unit  1 123-A3-23 
(formerly  1 123-C2-58),  Formative  Evaluation  of  Integrated  Product  and  Process  Development 
(IPPD).  This  research  was  conducted  under  Contract  No.  F41624-97-C-5012,  with  Metrica,  Inc. 
to  determine  how  current  IPPD  training  courses  can  best  be  adapted  in  order  to  be  suitable  for  Air 
Force  Research  Laboratory  (AFRL)  scientific  and  technology  (S&T)  efforts.  The  Laboratory 
Contract  Monitor  (LCM)  was  Ms  Phyllis  Morse;  the  LCM's  Technical  Representative  was  Dr 
Robert  M.  Yadrick  (AFRL/HEA). 

Dr  Yadrick  is  currently  located  at  AFOMS/OMYO,  1550  Fifth  Street  East,  Randolph 
AFB  TX  78150-4449.  His  e-mail  address  is  Robert.Yardick@RANDOLPH.AF.MIL. 


AN  INTEGRATED  PRODUCT  AND  PROCESS  DEVELOPMENT 
PROGRAM  AND  RELATED  TRAINING  REQUIREMENTS: 
FORMATIVE  EVALUATION  AND  DEFINITION  FOR  AIR  FORCE 
LABORATORY  SCIENCE  AND  ENGINEERING  PERSONNEL 


INTRODUCTION 

Each  year  government  and  private  industry  in  the  United  States  spend  billions  of  dollars  on  training  and 
development  activities.  Wooldridge,  Baldwin,  and  Ford  (as  cited  in  Faerman  &  Ban,  1993)  estimate  that 
the  federal  government  alone  spends  at  least  $6,333  million  each  year  and  that  American  industries 
spend  up  to  $100  billion  on  training  and  development  activities.  Figures  like  these  inevitably  lead  some 
to  question  whether  this  money  is  being  well  spent.  This  is  a  difficult  question  to  answer  for  two 
reasons.  First,  there  is  surprisingly  little  systematic  evaluation  of  training  programs,  in  either 
government  or  the  private  sector,  which  actually  tries  to  test  the  linkage  between  training  and  changes  in 
work  behavior.  Second,  the  few  attempts  to  study  that  linkage  have  failed  to  show  positive  changes  in 
individual  and  organizational  behavior. 

Research  that  has  focused  on  evaluating  training  outcomes  often  argues  there  is  little  evidence  that  the 
dollar  spent  on  these  training  and  development  activities  actually  translate  into  changes  in  individuals’ 
behavior  on  the  job  or  improved  organizational  performance  (Baldwin  &  Ford,  1988).  Similarly, 
Newstrom  (1986)  argues  that  too  much  money  and  attention  are  spent  on  the  design  and  delivery  of 
programs  and  not  enough  on  efforts  to  increase  the  transfer  of  training  to  the  work  environment.  Over 
the  past  two  decades  organizations  in  both  the  government  and  private  sectors  have  been  asked  to 
become  more  accountable  and  to  justify  the  existence  of  programs  in  terms  of  cost  and  performance 
factors  (Faerman  &  Ban,  1993).  This  has  led  to  increased  interest  in  the  evaluation  of  training  programs. 

That  training  plays  a  crucial  role  in  U.S.  Air  Force  (USAF)  effectiveness  and  efficiency  is  indisputable. 
For  this  reason  the  USAF  dedicates  a  vast  amount  of  people  to  the  design  and  delivery  of  training. 
Moreover,  the  Air  Force  assigns  individuals  to  evaluate  training  so  that  training  decisions  can  be  made 
based  on  the  best  available  data.  In  fact,  according  to  Thorndike  (as  cited  in  Alliger  &  Tannenbaum, 
1996),  a  clear  understanding  of  what  kind  of  training  is  important  for  what  jobs,  and  what  factors 
influence  training  effectiveness  has  historically  been  an  emphasis  in  the  Air  Force. 

Background 

In  the  past  20  years,  the  Air  Force  and  other  organizations  have  used  large-scale  education  and  training 
programs  in  attempts  to  bring  about  significant  changes  in  organizational  cultures,  attitudes,  and 
operating  procedures.  Experience  with  such  high  profile  programs  as  Management  By  Objectives 
(MBO),  multicultural/diversity  awareness  (including  Equal  Employment  Opportunity  [EEO]  and  sexual 
harassment  training),  and  Total  Quality  Management  (TQM)  taught  us  that  even  the  most  elaborate  and 
well  -thought-out  education  and  training  efforts  require  tailoring  and  considerable  support  with  an 
organization  to  bring  about  the  intended  outcomes.  Lessons  learned  from  these  programs  reinforce  the 
need  for  well-planned  evaluation  and  feedback  mechanisms  that  will  assess  outcome  achievement  and 
facilitate  timely  program  refinements. 
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Some  of  these  lessons  learned  include: 


•  one  size  does  not  fit  all, 

•  attitude  is  as  important  as  technical  skill, 

•  evaluation  of  outcomes  from  training  cannot  be  limited  to  measurement  of  knowledge  and  skills 
attained  at  the  end  of  the  course, 

•  nontraining  components  need  to  be  addressed  to  determine  if  obstacles  within  the  job  environment 
hindered  effective  application  or  use  of  principles, 

•  the  “course”  is  never  over,  and 

•  the  evaluation  should  be  consistent  with  the  spirit  of  the  course. 

These  lessons  were  used  to  guide  the  development  of  ideas  to  include  in  the  training  evaluation  surveys 
and  interviews  for  this  project. 

Development  and  implementation  of  Science  and  Technology  (S&T)  Integrated  Process  and  Product 
Development  (IPPD)  training  is  aimed  at  providing  the  Air  Force  S&T  community  with  the  knowledge, 
skills,  ability,  and  motivation  to  support  and  adapt  to  changing  customer  requirements  for  technology 
transition.  Air  Force  Materiel  Command  (AFMC)  believes  the  IPPD  process  represents  “an  excellent 
method  to  systematically  manage  priorities,  performance,  cost  and  risk,”  and  that  learning  to  use  this 
process  will  help  S&T  members  operate  more  effectively  in  a  changing  technology  transfer  environment 
Traditionally,  however,  real-world  application  of  IPPD  principles  and  methods  were  largely  confined  to 
commercial  manufacturing  settings.  It  was  not  clear  how  to  adapt  IPPD  principles  and  methods  to  Air 
Force  S&T  projects,  or  whether  IPPD  principles  and  methods  would  prove  to  be  equally  applicable  to  or 
beneficial  across  a  broad  spectrum  of  differing  types  of  Air  Force  S&T  projects. 

To  help  answer  these  questions,  the  Air  Force  Research  Laboratory  (AFRL)  designed  a  test  program  to 
implement  IPPD  procedures  in  several  6.3  Advanced  Technology  Demonstration  (ATD)  projects  in  each 
of  four  Air  Force  research  laboratories.  They  contracted  with  Texas  Instruments  Learning  Institute 
(TTLI)  and  James  Gregory  Associates  to  provide  initial  training  and  follow-up  mentoring  that  would 
enable  the  project  Integrated  Product  Teams  (IPTs)  to  implement  the  IPPD  procedures.  The  original 
training  provided  was  a  Design  for  Six  Sigma  Manufacturing  (DSSM)  course  that  TILI  had  used 
successfully  to  train  IPPD  processes  in  a  manufacturing  setting.  Based  on  trainee  feedback,  this  course 
was  adapted  into  the  Affordable  Technology  through  IPPD  (AT/IPPD)  course,  oriented  more  for  the 
S&T  community.  Additionally,  IPT  members  had  the  option  of  attending  additional  IPPD-related 
training  courses,  and  experts  experienced  with  applying  IPPD  in  industrial  settings  were  available  for 
“mentoring”  IPT  members,  regarding  application  of  IPPD  to  their  specific  projects. 

A  systematic  formative  evaluation  program,  including  feedback  concerning  the  effectiveness  when 
applying  the  IPPD  initiative  to  Air  Force  S&T  projects,  was  initiated  to  (a)  indicate  the  extent  to  which 
pilot  teams  are  able  to  effectively  apply  IPPD  methods  to  S&T  projects,  and  (b)  describe  how  IPPD 
methods  or  training  could  best  be  adapted  in  order  to  be  more  effectively  applied  to  S&T  projects.  Other 
issues,  such  as  the  role  of  ancillary  training  courses  and  mentoring  interaction  (e.g.,  which  courses 
are/were  most  useful,  to  what  extent  did  pilots  use  the  mentoring  opportunities,  what  effect  did  the 
mentoring  have)  were  also  addressed.  As  a  formative  evaluation,  this  research  program  was  not  designed 
to  provide  a  definitive  final  statement  of  the  utility  of  IPPD  training  and  implementation.  Rather,  it  was 
designed  to  provide  both  a  review  of  the  initial  training  experience  and  feedback  concerning  its  utility  in 
the  IPPD  pilot  projects  to  date.  Thus,  training  consultants  and  IPPD  implementation  managers  have  a 
basis  for  determining  how  to  improve  the  entire  IPPD  educational  experience  and  the  support  required  in 
the  S&T  laboratory  community  to  maximize  its  effectiveness. 
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Purpose  of  the  Research 


The  Department  of  Defense  is  responsible  for  providing  the  most  advanced  warfighting  capability  at  the 
most  affordable  cost.  This  responsibility  is  a  constant  but,  in  a  time  of  budget  decreases  and  downsizing, 
the  scope  of  capabilities  that  can  be  provided  to  the  warfighter  is  acutely  dependent  on  controlling 
acquisition  and  life  cycle  cost.  Attention  to  affordability  must  occur  seamlessly  throughout  the  weapons 
development  cycle,  particularly  in  the  S&T  phase  where  it  has  been  traditionally  neglected  in  favor  of 
performance.  Manufacturing  and  business  processes  developed  in  S&T  must  be  sufficiently  matured  that 
they  may  transition  to  demonstration/validation,  or  engineering  and  manufacturing  development  without 
the  need  for  costly  reinvention  in  terms  of  both  time  and  money  (P.G.  Kaminski,  personal 
communication,  August  7,  1996). 

The  affordability  issue  is  a  Department  of  Defense  initiative.  The  Director,  Defense  Research  and 
Engineering  (DDR&E),  chartered  an  S&T  Affordability  Task  Force  in  May  1995  to  address  the  issues  of 
assessing  and  strengthening  the  affordability  focus  of  military  S&T  programs.  Findings  from  that  task 
force  identified  a  set  of  best  practices  for  evaluating  an  S&T  project  with  attention  towards  affordability, 
(i.e.,  does  the  project  have  exit  criteria,  have  deliverables  been  clearly  defined,  is  Integrated  Product  and 
Process  Development  being  used,  etc.).  These  criteria  provide  the  basis  for  this  affordability  policy  (P.G. 
Kaminski,  personal  communication,  August  7, 1996). 

One  goal  of  this  research  effort  was  to  develop  and  implement  a  formative  evaluation  program  that 
would  help  determine  the  extent  to  which  the  AT/IPPD  training  program  has  facilitated  employment  of 
the  IPPD  process  in  the  pilot  IPTs  that  received  that  training.  Other  goals  include  identifying  obstacles 
that  mitigate  against  using  IPPD  procedures  in  the  S&T  community  and  providing  observations  in  which 
that  training  and  the  climate  for  its  use  could  be  improved  and  lead  to  a  more  effective  implementation  of 
the  DOD  affordability  policies. 

Specific  objectives  related  to  these  goals  were  as  follows: 

1 .  Gather  and  analyze  data  regarding  the  effectiveness  of  the  IPPD  training  program  for  Air  Force 
S&T  pilot  project  personnel. 

2.  Identify  obstacles  to  successful  implementation  of  the  IPPD  initiative  in  Air  Force  laboratories, 
including  problems  or  deficiencies  in  the  training  program,  inherent  incompatibilities  between  the  IPPD 
concept  and  the  nature  of  Air  Force  laboratory  work  and  projects,  and  such  “cultural”  obstacles  as  lack  of 
management  support  for  the  initiative  or  lack  of  acceptance  of  IPPD  principles  by  project  personnel. 

3.  Compare  projects  from  within  the  same  laboratory  and  across  different  laboratories  to  identify 
differing  implementation  processes  or  obstacles  that  result  across  project  types  or  laboratory 
environments. 

4.  Assess  the  potential  for  employing  alternative  means  of  delivery  such  as  computer-aided 
instruction  or  distance  learning  techniques. 

5.  Provide  recommendations  concerning  instruments  and  methods  for  a  long-term  program 
evaluation  effort. 


3 


The  criteria  for  assuring  a  focus  on  affordability  in  S&T  are  that: 

•  customer  needs  are  understood, 

•  management  is  willing  and  able  to  support  IPPD, 

•  program  strategy  addresses  critical  affordability  issues, 

•  affordability  metrics  have  been  established  and  are  being  measured,  and 

•  a  training  plan  has  been  developed  and  executed. 


METHODOLOGY 
The  Research  Plan 

The  general  research  plan  involved  two  related  data  collection  efforts.  First,  researchers  would  observe 
the  AT/IPPD  training  sessions,  gather  feedback  through  discussions  with  participants  and  an  end-of- 
course  questionnaire,  and  provide  the  training  team  with  interim  summaries  of  participant  responses  and 
recommendations  for  improving  course  content  or  delivery.  These  data  were  designed  to  determine  the 
participants’  perceptions  of  their  entering  knowledge  of  and  attitude  about  the  IPPD  process  and  the 
extent  to  which  their  understanding  and  attitude  changed  by  the  end  of  the  three-day  training  session. 

The  second,  more  difficult  data  collection  effort,  involved  gathering  follow-up  information  from  IPT 
members  who  had  taken  IPPD  training  at  least  six  months  previously,  about  the  extent  to  which  they 
were  able  to  implement  those  procedures  in  their  pilot  projects.  Data  collection  required  visits  to  several 
AFRL  laboratories  and  using  small  group  and/or  telephone  interviews.  Questionnaires  were  also  used  to 
obtain  feedback  from  those  interviewed  and  from  team  members  who  had  participated  in  training  but 
were  not  available  for  the  interview  sessions.  In  conjunction  with  the  evaluation  project’s  government 
program  monitor  (GPM),  researchers  agreed  to  gather  data  from  members  of  several  IPTs  who  had  not 
taken  the  AT/IPPD  course,  but  who  had  participated  in  the  DSSM  training  course  at  TILI  between  May 
and  November  1996.  That  course  had  served  as  a  basis  for  the  eventual  AT/IPPD  course,  which  was 
presented  over  seven  sessions  between  April  and  October  1997.  Table  1  is  a  summary  of  the  13  IPTs  and 
the  numbers  who  had  attended  either  the  DSSM  or  IPPD  course.  The  total  of  56  DSSM  participants 
reflects  four  individuals  who  attended  both  the  DSSM  course  in  1996  and  an  AT/IPPD  session  in  1997. 

In  addition,  discussions  with  pilot  members  at  the  first  training  sessions  led  to  the  realization  that  it 
would  be  difficult,  in  the  time  allotted  in  the  original  contract,  to  assess  the  implementation  of  IPPD 
among  those  pilots  whose  members  took  the  training  after  June  1997.  There  would  not  be  sufficient 
time  for  some  of  those  pilots  to  employ  the  IPPD  procedures  by  the  time  data  gathering  would  have  to  be 
completed  in  November  1997.  Therefore,  the  GPM  and  the  researchers  agreed  to  a  six-month,  no-cost 
extension  of  the  contract  so  data  could  be  gathered  through  March  1998  and  more  of  the  pilot  teams 
could  be  included. 

Because  the  subject  sample,  data  collection  devices  and  the  purpose  of  the  two  phases  of  the  research 
effort  are  different,  the  phases  will  be  dealt  with  separately  throughout  the  Methodology  and  Results 
sections  of  this  report.  In  each  section,  the  information  pertaining  to  the  immediate  impact  of  the 
AT/IPPD  training  course  will  be  provided.  Then,  the  information  related  to  the  actual  implementation  of 
the  IPPD  process  in  the  various  pilot  programs  will  be  presented. 
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Table  1.  DSSM/IPPD  Training  Session  Participants  by  Pilot  Project 


PILOT  PROJECT  NUMBER  OF 

PARTICIPANTS 


DSSM 

IPPD 

3-D  Optical  Computer  Memory  Storage  (3D-OP) 

9 

Advanced  Motor  Drive  (AMD) 

3 

25 

Advanced  Laser  Eye  Protection  (ALEP) 

Composite  Affordability  Initiative  (CAI) 

15 

29 

Large  Aircraft  Infrared  Countermeasures  (IRCM) 

Integrated  Space  Technology  Demonstration  (ISTD) 

6 

Improved  Space  Computer  Program  (ISCP) 

5 

1 

Low  Cost  Autonomous  Attack  System  (LCAAS) 

Modular  Aircraft  Support  System  (MASS) 

1* 

7 

Next  Generation  Transparency  (NGT) 

20* 

7 

Over  The  Horizon  Technologies  (OTH) 

7 

Virtual  Interactive  Intelligent  Tutoring  System  Development  Shell 
VIVIDS) 

12 

7 

Ballistic  Winds  Airdrop  (WINDS) 

3 

TOTAL  56*  101 

*  4  members  attended  both  DSSM  and  AT/IPPD  training _ 


Analysis  of  the  AT/IPPD  Training  Sessions 

Data  Collection  Instrument/Methods.  Project  researchers  attended  all  seven  of  the  AT/IPPD  training 
sessions.  Because  the  research  contract  did  not  begin  until  May  1997,  the  GPM  attended  the  first  trial 
session  of  the  new  course  in  April,  which  had  only  five  participants,  and  only  three  from  an  IPT.  He 
administered  a  preliminary  42-item  questionnaire  that  served  as  the  basis  for  the  instrument  used  in  the 
other  six  sessions.  The  questionnaires  administered  to  the  participants  of  the  other  six  training  sessions 
contained  54  items  and  subitems  and  differed  primarily  by  the  anchoring  of  intermediate  points  in  a  few 
of  the  scales  and  by  addition  of  seven  questions  asking  about  potential  obstacles  to  implementation  of  the 
IPPD  process  in  their  projects.  While  a  majority  of  the  questions  had  respondents  select  from  a  5-point 
modified  Likert  scale,  several  demographic  questions  required  selection  from  a  short  list  of  options.  A 
number  of  questions  related  to  the  course  content  offered  open-ended  opportunities  for  comment  or 
recommendations.  A  sample  of  this  questionnaire  is  found  in  Appendix  A. 

Table  2.  Number  Responding  to  IPPD  Training  Session  Evaluation  Questionnaire  by  Pilot  Project. 


_ PILOT  PROJECT  TITLE _ RESPONDENTS 

3-D  Optical  Computer  Memory  Storage  (3D-OP)  9 

Advanced  Laser  Eye  Protection  (ALEP)  25 

Composite  Affordability  Initiative  (CAI)  29 

Integrated  Space  Technology  Demonstration  (ISTD)  5 

Low  Cost  Autonomous  Attack  System  (LCAAS)  1 

Modular  Aircraft  Support  System  (MASS)  7 

Next  Generation  Transparency  (NGT)  7 

Over  The  Horizon  Technologies  (OTH)  7 

Virtual  Interactive  Intelligent  Tutoring  System  Development  Shell  (VIVIDS)  7 

Ballistic  Winds  Airdrop  (WINDS) _ 3 _ 

TOTAL  100 
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Subject  Sample.  Of  1 16  participants  in  the  seven  IPPD  training  sessions,  102  were  associated  with  one  of 

the  10  IPTs  represented  at  those  sessions.  Questionnaire  responses  were  received  from  100  of  those  participants 
(Table  2),  representing  both  government  and  contractor  personnel,  and  ranging  across  a  number  of  project  positions 
(see  Tables  3  and  4).  Because  a  larger  number  of  participants  were  from  two  of  the  pilot  projects,  researchers 
continuously  compared  their  responses  to  those  of  other  pilots  to  determine  if  their  input  skewed  the  overall  results. 

Table  3.  IPPD  Training  Session  Questionnaire  Respondents  by  Organization 

ORGANIZATION  RESPONDENTS 

Armstrong  Laboratory 

12 

Philips  Laboratory 

7 

Rome  Laboratory 

12 

Wright  Laboratory 

13 

AD  Little 

3 

Anteon 

2 

Boeing 

11 

Call/Recall 

1 

Command  Technology 

1 

Coherent  Technology 

3 

Dalloz  Safety 

2 

GE  Aircraft 

1 

KOSI 

2 

LAE 

1 

Lockheed 

4 

MANTECH 

1 

MTC 

1 

NAVAIR 

5 

Northrop 

3 

Pilkington  OP 

1 

Rockwell 

1 

Synectics 

1 

TASC 

8 

use 

1 

TOTAL 

100 

Table  4.  IPPD  Training  Session  Questionnaire  Respondents  by  Position. 

POSITION 

RESPONDENTS 

Administrative  Manager 

4 

Administrative  Supervisor 

1 

Budget  Manager 

1 

Consultant 

2 

Cost  Analyst 

3 

Engineer 

18 

Integrated  Product  Team  Leader 

3 

Program  Manager 

5 

Researcher 

8 

Scientist 

3 

Technical  Assistant 

1 

Technical  Manager 

37 

Technical  Monitor 

2 

Technical  Specialist 

1 

Technical  Supervisor 

7 

TOTAL 

100 
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Data  Analysis.  The  questionnaire  data  were,  initially,  entered  into  Excel  spreadsheets.  The  completed 
database  was  then  saved  as  a  flat-text  file.  The  flat  file  was  imported  into  the  Stata  statistical  package 
which  was  used  to  conduct  all  the  statistical  processing  and  analyses  (StataCorp,  1997).  The  write-in 
comments  from  each  questionnaire  were  summarized  on  a  spreadsheet  for  qualitative  analysis  to 
determine  patterns  of  responses. 

Analysis  of  the  Implementation  of  IPPD  Procedures  in  the  IPT  Pilots 

Data  Collection  Instruments/Methods.  Information  regarding  the  trainees’  ability  to  actually 
implement  the  IPPD  process  in  their  pilot  projects  was  gathered  through  two  methods.  First,  structured 
interviews  were  undertaken  with  the  program  managers  and,  where  possible,  key  members  of  the  IPTs  at 
least  six  months  after  they  had  participated  in  the  IPPD  training.  This  was  the  minimal  amount  of  time 
that  was  considered  necessary  to  enable  the  teams  to  show  any  progress  in  the  use  of  IPPD  procedures. 
The  interview  items  were  developed  from  discussions  with  IPT  members  at  the  training  sessions  and 
were  tested  and  refined  at  a  preliminary  session  at  Wright  Laboratory  in  August.  The  list  of  interview 
session  discussion  questions  is  found  in  Appendix  B. 

In  addition  to  structured  interviews,  we  gathered  additional  supportive  information  via  a  Follow-up 
Questionnaire  administered  to  each  of  the  interview  session  participants  and  mailed  to  members  of  the 
IPT  that  had  attended  IPPD  or  DSSM  training  but  were  not  available  for  interviews.  This  questionnaire 
once  again  was  developed  based  on  discussions  with  IPT  members  at  the  training  sessions,  with  IPPD 
program  managers  at  AFMC,  and  with  thq  training  consultants.  It  was  pilot  tested  among  several  IPT 
members  at  Brooks  AFB  and  was  refined  to  insure  adequate  coverage  of  issues  as  well  as  ease  of 
understanding  by  participants.  The  instrument  consisted  of  62  items  or  subitems  and  included  a  few 
demographic  description  questions,  46  questions  employing  a  5-point  Likert  scale,  and  a  few  open-ended 
questions  allowing  for  write-in  comments.  A  number  of  the  questions  were  similar  to  those  on  the 
training  session  questionnaire  (e.g.,  regarding  participants’  knowledge  of  and  ability  to  apply  IPPD, 
obstacles  to  implementation,  and  perceptions  of  management  knowledge  and  attitude  about  IPPD). 
However,  these  were  intended  to  get  a  picture  of  IPT  members’  impressions  after  they  had  been  back  on 
their  projects  and  had  had  a  chance  to  implement  the  IPPD  procedures  in  a  work  environment. 

Additional  questions  focused  on  the  extent  to  which  individuals  and  IPT  teams  have  actually  employed 
various  IPPD  processes  in  their  work  efforts  and  the  extent  to  which  use  of  IPPD  methods  seemed  to 
help  the  projects.  A  sample  of  the  Follow-up  Questionnaire  is  found  in  Appendix  C. 

Finally,  it  became  apparent  from  the  group  interviews  in  November  1997  that  the  IPPD  implementation 
was  an  ongoing  process  and  that  additional  application  would  occur  in  several  projects  in  the  four 
months  (December  1997  through  March  1998)  after  the  initial  interview  and  Follow-up  Questionnaire 
data  were  gathered.  In  an  effort  to  provide  the  most  current  data,  follow-up  telephone  and  personal 
interview  sessions  were  conducted  in  late  March  and  April  with  program  managers  from  nine  pilots 
contacted  earlier  and  with  two  pilots  that  had  taken  the  IPPD  course  in  August  and  September  1997.  At 
the  same  time,  to  provide  a  more  common  metric  for  updating  and  summarizing  the  status  of  the  pilots, 
program  managers  filled  out  a  short  Summary  of  IPPD  Processes  Used  survey.  This  instrument  asked 
them  to  describe  their  project’s  experience  with  each  of  six  aspects  of  the  IPPD  process  by  selecting 
from  a  series  of  descriptive  choices  derived  from  the  researchers’  previous  data  gathering.  A  copy  of 
that  survey  is  found  in  Appendix  D. 
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Subject  Sample.  Of  the  13  IPT  projects  initially  intended  as  IPPD  pilots,  two  encountered  unforeseen 
difficulties  that  precluded  their  continuation  as  pilots.  The  Low  Cost  Autonomous  Attack  System  project 
was  delayed,  and  only  one  member  participated  in  the  IPPD  training.  The  Integrated  Space  Technology 
Demonstration  program  was  unable  to  continue  as  an  IPPD  pilot  because  of  contractual  difficulties. 

Little  evaluative  information  was  obtained  from  these  two  projects,  and  researchers  decided  to  include 
only  the  results  of  the  other  1 1  pilots. 

Researchers  made  follow-on  visits  to  AFRL  sites  at  Wright  Patterson  AFB  in  Ohio;  Rome  Laboratory, 
New  York;  and  Brooks  AFB,  Texas,  where  they  interviewed  59  team  members  in  small  groups  from  10 
of  the  13  original  IPTs.  Telephone  interviews  were  employed  with  the  program  manager  of  the  ISCP 
project  at  Kirtland  AFB,  and  limited  telephone  contact  was  made  with  representatives  of  two  of  the  IPTs 
whose  participation  as  IPPD  pilots  was  interrupted  by  unforeseen  circumstances.  A  summary  of  the  IPTs 
teams  interviewed  is  at  Table  5. 

Follow-up  Questionnaires  were  administered  to  IPT  members  who  attended  the  IPPD  or  earlier  DSSM 
training  sessions.  They  were  distributed  either  at  the  interview  sessions,  through  program  managers,  or 
mailed  directly  to  the  subjects  at  their  work  sites  after  they  had  first  been  notified  by  their  program 
managers.  After  three  weeks,  follow-up  appeals  were  sent  to  nonrespondents  either  by  the  program 
representatives  or  directly  through  electronic  or  telephone  messages.  Of  the  original  154  IPPD  or  DSSM 
training  participants,  27  were  not  included  in  the  administration  because  they  were  no  longer  associated 
with  the  pilot  project.  Overall  return  rate  was  68  %;  the  rate  was  above  65%  for  8  of  the  1 1  IPTs  that 
received  questionnaires.  Table  5  summarizes  the  questionnaires  sent  and  subjects  responding. 


Table  5.  Members  Participating  in  IPPD  Follow-up  Data  Gathering  by  IPT 


PILOT  PROJECTS 

INTERVIEWS 

QUESTIONNAIRES 

SENT/RESPONSES 

3-D  Optical  Computer  Memory  Storage  (3D-OP) 

6 

9/6 

Advanced  Motor  Drive  (AMD) 

2 

3/3 

Advanced  Laser  Eye  Protection  (ALEP) 

12 

24/16 

Composite  Affordability  Initiative  (CAI) 

6 

29/16 

Large  Aircraft  Infrared  Countermeasures  (IRCM) 

12 

14/11 

Improved  Space  Computer  Program  (ISCP) 

1 

5/3 

Modular  Aircraft  Support  System  (MASS) 

3 

7/5 

Next  Generation  Transparency  (NGT) 

4 

14/9 

Over  The  Horizon  Technologies  (OTH) 

6 

7/6 

Virtual  Interactive  Intelligent  Tutoring  System 
Development  Shell  (VIVIDS) 

5 

7/5 

Ballistic  Winds  Airdrop  (WINDS) 

3 

8/4 

TOTAL 

60 

127/86 

Data  Analysis.  As  with  the  post-training  questionnaire  data,  the  Follow-up  Questionnaire  data 
were  also  entered  into  Excel  spreadsheets  and  then  saved  as  a  flat-text  file.  The  flat  file  was  imported 
into  the  STATA  statistical  package  that  was  used  to  conduct  all  the  statistical  processing  and  analysis. 
Write-in  comments  from  each  questionnaire  were  copied  and  summarized  on  a  spreadsheet  for 
qualitative  analysis  to  determine  patterns  of  responses.  The  interview  notes  were  summarized  after  each 
session  and  were  reviewed  with  the  program  managers  to  ensure  there  was  no  confusion  regarding 
terminology  or  status/direction  of  the  projects.  Researchers  compared  the  project  summary  impressions 
with  questionnaire  responses  for  the  IPTs  to  obtain  a  more  complete  analysis  of  the  IPPD 
implementation  situation. 
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RESULTS 


AT/EPPD  Training  Sessions 

Perceptions  of  General  IPPD  Knowledge/Understanding.  The  majority  (51%)  of  the  participants 
reported  entering  the  course  with  none  to  a  small  amount  of  IPPD  experience,  while  only  23%  indicated 
having  a  large  to  very  large  amount  of  IPPD  experience.  However,  two  pilot  teams  reported  having  a 
much  greater  amount  of  IPPD  experience.  As  one  of  them  had  a  large  number  of  respondents,  that 
pilot’s  responses  somewhat  skewed  the  results  to  that  item.  If  that  pilot’s  responses  had  been  normalized 
to  reflect  the  size  of  most  other  groups,  the  number  reporting  none  or  a  small  amount  of  experience 
would  have  been  61%,  rather  than  51%.  Across  the  general  results  from  the  training  participants’ 
questionnaire,  this  was  the  only  area  in  which  the  responses  from  one  of  the  two  large  pilot  groups 
affected  the  overall  results  by  more  than  9  percentage  points.  The  by-pilot  breakout  of  responses  to  this 
and  many  other  items  are  found  in  Appendix  E. 

With  regard  to  prior  IPPD  knowledge,  initially  58%  indicated  they  entered  training  with  none  to  a  small 
amount  of  knowledge.  After  the  training,  the  percentage  indicating  they  presently  have  none  to  a  small 
amount  of  knowledge  dropped  dramatically  to  8%  (see  Table  6). 

Table  6.  Percentage  of  Respondents  Reporting  Various  Amounts  of  IPPD  Experience  and  Knowledge 


NONE  TO  SMALL 

51% 

PRIOR  IPPD  EXPERIENCE 
MODERATE 

26% 

LARGE  TO  VERY  LARGE 

23% 

NONE  TO  SMALL 

PRIOR  IPPD  KNOWLEDGE 

MODERATE 

LARGE  TO  VERY  LARGE 

58% 

28% 

14% 

IPPD  KNOWLEDGE  AFTER  TRAINING 

SMALL 

MODERATE 

LARGE  TO  VERY  LARGE 

8% 

51% 

41% 

Fairly  encouraging  changes  were  indicated  in  perceived  understanding  of  IPPD  initiative,  benefits,  and 
application  following  course  completion.  In  terms  of  prior  understanding  of  the  IPPD  initiative,  25% 
reported  good  or  excellent  understanding  of  reasons  for  initiative— that  percentage  increased  to  64%  after 
training.  With  regard  to  IPPD  benefits,  24%  reported  entering  training  with  good  or  excellent 
understanding;  this  percentage  increased  to  60%  after  training.  Furthermore,  while  69%  reported  none  to 
slight  understanding  of  the  IPPD  application  to  projects,  that  percentage  dropped  to  19%  upon 
completion  of  training  (see  Table  7). 

Perceptions  of  Ability  to  Apply  EPPD  Procedures.  Relatively  few  reported  strong  confidence  in  their 
overall  ability  to  apply  training  after  the  initial  IPPD  course.  While  29%  said  they  were  capable  or  very 
capable,  46%  said  they  were  only  moderately  capable,  and  25%  said  they  were  slightly  capable  or  not  at 
all.  Additionally,  the  majority  of  respondents  (54%)  reported  they  needed  at  least  a  moderate  amount  of 
additional  training,  while  28%  said  they  needed  a  large  or  very  large  amount  of  additional  training  (see 
Figures  1  and  2). 


9 


Table  7.  Percentage  of  Respondents  Reporting  Amount  of  IPPD  Understanding  Before  and  After  IPPD 
Training 


UNDERSTANDING  INITIATIVE 

NONE  TO  SLIGHT 

MODERATE 

GOOD  TO  EXCELLENT 

PRIOR 

50% 

25% 

25% 

POST  TRAINING 

4% 

32% 

64% 

UNDERSTANDING  BENEFITS 

NONE  TO  SLIGHT 

MODERATE 

GOOD  TO  EXCELLENT 

PRIOR 

56% 

20% 

24% 

POST  TRAINING 

5% 

34% 

60% 

UNDERSTANDING  APPLICATION 

NONE  TO  SLIGHT 

MODERATE 

GOOD  TO  EXCELLENT 

PRIOR 

69% 

21% 

10% 

POST  TRAINING 

19% 

41% 

40% 

Attitudes  about  Potential  Application  and  Utility  of  IPPD.  There  were  some  changes  in  projections 
for  success  of  IPPD,  although  a  fairly  large  group  of  doubters  were  still  represented.  Originally,  only 
19%  believed  their  ability  to  successfully  apply  IPPD  to  their  project  was  likely  or  highly  likely,  but  after 
training,  that  percentage  increased  to  43%.  However,  after  training,  17%  reported  it  was  unlikely  or 
highly  unlikely  that  they  could  successfully  apply  the  IPPD  techniques  to  their  projects  (see  Figures  3 
and  4). 

With  regard  to  IPPD  providing  real  benefits  to  their  pilot  projects,  18%  originally  reported  IPPD  was 
likely  to  provide  real  benefits,  but  after  training,  40%  of  respondents  reported  that  IPPD  was  likely  or 
highly  likely  to  provide  real  benefits  to  their  projects.  After  training,  only  16%  believed  it  was  unlikely 
or  highly  unlikely  that  IPPD  would  provide  real  benefits  (see  Figures  5  and  6). 

Respondents  reported  fairly  strong  perceptions  of  general  utility  of  the  IPPD  process  and  training  beyond 
their  current  pilot  project.  The  respondents’  opinions  were  split  with  regard  to  IPPD  having  benefits  for 
other  6.3  S&T  projects— 48%  perceived  moderate  benefit  to  other  projects,  while  48%  saw  large  or  very 
large  benefits  to  using  the  IPPD  process.  In  terms  of  the  IPPD  process  being  useful  for  future 
professional  development,  30%  said  the  IPPD  process  was  moderately  useful,  but  61%  perceived  the 
process  as  quite  or  extremely  useful  (see  Figures  7  and  8). 

Perceptions  of  Management’s  Understanding  of/Belief  in  IPPD.  Relatively  few  respondents 
perceived  a  large  amount  of  knowledge  or  faith  in  the  IPPD  process  among  management.  In  terms  of 
management’s  knowledge  of  the  IPPD  process,  8%  perceived  their  knowledge  was  large  or  veiy  large; 
however,  57%  said  none  or  small,  and  7%  didn’t  know.  When  asked  about  their  management’s 
understanding  of  reasons  for  IPPD,  21%  indicated  their  understanding  was  good  or  excellent;  however, 
46%  said  none  or  slight,  and  8%  didn’t  know.  The  results  were  similar  with  regard  to  management’s 
understanding  of  IPPD  benefits— 21%  reported  perceiving  their  understanding  as  good  or  excellent,  but 
again,  43%  said  none  or  slight,  and  9%  didn’t  know  (see  Figures  9, 10,  and  1 1). 


■  NOT  CAPABLE 
■SLIGHTLY 
□  MODERATELY 
■CAPABLE  * 
■VERY  CAPABLE 


47% 


Figure  1.  Capability  of  IPPD  Application  after  Training. 


Figure  2.  IPPD  Additional  Training  or  Mentoring  Needed 
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Figure  6.  Post-training  Projections  about  IPPD  Achieving  Rea!  Benefits 
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Figure  7.  IPPD  Benefits  to  Other  Projects 
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Figure  9.  Perception  of  Management's  Knowledge  of  IPPD  Process 
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Figure  10.  Perception  of  Management's  Understanding  of  IPPD  Initiative 
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Figure  11.  Perception  of  Management's  Understanding  of  IPPD  Benefits 
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Continuing  the  trend,  respondents  reported  similar  results  for  their  management’s  understanding  of  IPPD 
application  to  their  pilot  projects  (13%  said  good  or  excellent;  58%  said  none  or  slight;  13%  didn’t 
know);  their  management’s  projections  for  successful  application  (20%  said  likely  or  highly  likely,  18% 
indicated  unlikely  or  highly  unlikely,  and  32%  didn’t  know);  as  well  as  their  management’s  projections 
for  the  IPPD  process  resulting  in  actual  benefits  (17%  indicated  likely  or  highly  likely,  17%  also 
indicated  unlikely  or  highly  unlikely,  and  34%  didn’t  know).  (See  Figures  12,  13,  and  14) 
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Figure  12.  Perception  of  Management’s  Understanding  of  IPPD  Application 
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Figure  13.  Perception  of  Management’s  Projections  about  Successful  IPPD  Application 
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Figure  14.  Perception  of  Management's  Projections  about  IPPD  Achieving  Real  Benefits 
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Perceptions  of  management  knowledge  and  understanding  tend  to  be  fairly  consistent  among  the  pilot 
groups.  When  compared  across  nine  of  the  pilot  projects  that  had  more  than  one  attendee,  the  responses 
to  the  knowledge  and  understanding  questions  generally  reveal  seven  or  eight  projects  with  less  than 
25%  of  the  respondents  indicating  a  good  to  excellent  understanding  on  the  part  of  management. 
However,  one  group  consistently  reported  greater  understanding  on  the  part  of  their  management,  with 
between  43%  and  57%  indicating  a  large  to  very  large  knowledge  or  good  to  excellent  understanding. 

There  was  somewhat  less  consistency  among  the  pilots  when  they  described  their  perceptions  of 
management’s  projections  for  the  likelihood  of  successfully  applying  IPPD  or  of  obtaining  benefits  from 
that  application.  For  these  two  questions,  more  than  40%  of  respondents  in  at  least  four  of  the  groups 
didn’t  know  what  their  management’s  projections  were.  In  seven  or  eight  of  the  pilot  groups  fewer  than 
33%  of  respondents  reported  that  their  management  perceives  IPPD  success  or  benefits  to  be  likely  or 
highly  likely.  Once  again,  there  is  that  same  group  with  57%  reporting  management  expects  that 
successful  IPPD  application  is  likely  or  highly  likely.  And  there  are  two  groups  in  which  43%  indicate 
their  management  thinks  it  is  likely  or  highly  likely  that  benefits  will  accrue  from  IPPD  application. 

Perceptions  of  Potential  Obstacles  to  Implementation.  Based  on  discussions  with  IPT  members  and 
training  consultants,  several  factors  consistently  identified  as  hindrances  to  IPPD  implementation  were 
listed  in  the  questionnaire.  Having  just  completed  the  training,  respondents  were  asked  to  use  a  5-point 
scale  and  rate  the  amount  that  each  factor  might  play  in  preventing  successful  application  of  IPPD  in 
their  project.  The  factor  identified  most  consistently  as  a  potential  obstacle  was  the  unknown  or 
changing  variables  or  conditions  associated  with  their  project  that  prevented  estimation  of  reliable 
parameters.  In  six  of  the  nine  pilot  projects  responding,  more  than  40%  of  eligible  respondents  indicated 
this  aspect  would  play  a  large  or  very  large  part  in  hindering  application  of  IPPD.  Table  8  shows  the 
prevention  impact  percentages  for  all  four  factors,  by  pilot  group. 

Table  8.  Percent,  by  Pilot,  Indicating  Various  Factors  Hindered  Their  IPPD  Implementation  a  Large  or  Very  Large 
Amount 


PILOT  TEAMS 


Hinderine  Factors 

I 

2 

3 

4 

5 

6 

7 

8 

9 

Too  many  unknown  or  changing  variables 

67 

40 

65 

0 

100 

43 

0 

43 

29 

Project  has  special  needs/qualities  not  to 
IPPD 

33 

16 

41 

40 

0 

0 

0 

86 

0 

Too  far  along  in  project  to  employ 

11 

0 

3 

20 

0 

0 

0 

86 

14 

Not  enough  time/Resources 

33 

40 

7 

20 

0 

0 

0 

43 

14 

Other 

0 

36 

29 

60 

0 

14 

14 

14 

57 

In  addition,  49  respondents  identified  factors  other  than  those  listed  and  indicated  they  could  have  a 
significant  impact  on  the  ability  to  implement  IPPD.  In  two  of  the  ten  IPTs,  more  than  40%  of  the 
eligible  respondents  indicated  “other”  factors  would  hinder  implementation  a  large  or  very  large  amount. 
The  most  common  of  those  additional  factors  were  insufficient  time,  lack  of  funding  for  IPPD  costs, 
management  commitment  to  IPPD,  difficulty  in  obtaining  user  involvement,  needed  practice  in  using 
IPPD  methods,  and  potential  budget  cuts. 
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Perceptions  of  Course  Content  and  Suggestions  for  Improving  the  Course.  It  is  evident  that 
participants  saw  personal  benefits  from  the  course  material,  since  as  shown  in  Figure  8,  60%  indicated  it 
is  quite  or  extremely  useful  for  their  professional  development.  These  respondents  also  saw  ways  in 
which  the  material  could  be  changed  to  better  prepare  them  for  IPPD  implementation  in  their  specific 
IPT  project.  Write-in  responses  to  open-ended  questions  about  the  content  and  conduct  of  the  IPPD 
course  echoed  many  of  the  perceptions  expressed  to  researchers  during  the  various  sessions  and  provided 
excellent  feedback  for  course  improvement.  A  short  summary  of  the  responses  to  each  of  these 
questions  follows. 

1.  Course  Content  Most  Useful  or  Relevant:  When  asked  to  identify  the  block  of  instruction  they 
perceived  as  most  useful  or  relevant,  94  of  the  participants  provided  a  response,  and  77  of  those  listed 
reasons  for  their  selection.  A  large  majority  of  respondents  (74)  indicated  that  either  the  House  of 
Quality  (39)  or  Design  of  Experiments  (35)  modules  were  most  useful.  The  sessions  on  Monte  Carlo 
Technique,  Equation-Based  Analysis,  and  Value  Scorecards  were  cited  a  total  of  8,  7,  and  5  times 
respectively. 

The  reasons  offered  for  usefulness  or  relevance  varied.  Respondents  citing  the  House  of  Quality  as  most 
useful  indicated  this  technique  allows  identification  of  customer  and  engineering  requirements  up  front, 
and  they  explained  that  it  forces  the  user  to  be  more  organized,  gathering  only  useful  data,  related  to 
requirements.  Those  citing  the  Design  of  Experiments  as  most  useful  reported  that  technique  allows  the 
user  the  capability  of  statistically  sorting  and  evaluating  critical  requirements  based  on  defined 
requirements  and  tolerance.  They  indicated  that  the  IPPD  system  provides  the  opportunity  to  effectively 
rate  different  solutions  and  to  allow  outside  evaluations  of  pending  decisions.  A  few  respondents 
reiterated  that  the  above  techniques  would  be  applied  to  their  individual  pilot  projects. 

2.  Course  Content  Least  Useful  or  Relevant :  Fewer  (69)  of  the  respondents  offered  comments  about 
their  least  favorite  blocks  of  instruction— and  these  were  more  widely  varied.  Ten  of  the  respondents 
cited  the  Unmanned  Research  Vehicle  exercise  as  the  least  useful  portion  of  the  course,  indicating  there 
wasn’t  enough  time  to  complete  the  different  parts  of  the  exercise,  that  too  much  time  was  spent  “number 
crunching,”  or,  most  commonly,  that  the  exercise  related  more  to  production  or  manufacturing  processes 
and  not  to  S&T.  Other  modules  cited  as  least  useful  or  relevant  were  as  follows: 

•  Design  of  Experiments  (8  times) 

•  Equation-Based  Analysis  (6  times) 

•  Monte  Carlo  (6  times) 

•  Expert  Judgement  (3  times) 

•  House  of  Quality  (2  times) 

A  common  concern  (of  those  who  cited  a  least  useful  block)  was  that  the  mathematical  demands  of  the 
course  were  either  too  confusing,  presented  in  too  much  detail,  or  not  relevant  to  the  overall  under¬ 
standing  of  the  IPPD  process.  Several  comments  indicated  the  material  was  too  complex/particular  for 
the  audience  (i.e.,  program  managers  don’t  have  to  be  statisticians  to  manage  programs  using  IPPD). 
Forty  respondents  provided  suggestions  to  make  these  sections  more  useful  and  included  the  following: 

increase  the  course  time  allotted  or  decrease  the  URV  exercise  and  mathematical  requirements 
provide  software  to  assist  with  different  module  tasks,  rather  than  making  everyone  crunch  the 
numbers  manually 

tailor  the  course  modules  more  to  the  specific  pilot  projects  or  more  effectively  show  the 
application  of  the  principles  to  6.3  S&T  activities 
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3.  Suggestions  for  Improving  the  Course:  The  last  two  items  on  the  questionnaire  asked 
respondents  if  they  would  make  any  other  changes  to  the  course  content  or  conduct  and  solicited  specific 
suggestions.  Responses  from  71  participants  were  varied,  but  echoed  many  of  the  earlier  suggestions. 

-  better  organize  or  execute  the  course  exercise;  the  idea  was  good,  but  the  execution  and  instructions 
left  some  work  teams  confused 

-  develop  an  exercise  more  in  line  with  pilot  projects 

-  show  better  application  of  the  process  and  techniques  to  6.3  S&T  activities 

-  spend  less  time  on  mathematics  or  provide  teams  with  a  computer  and  spreadsheet  software  to  do  the 
math  so  more  time  could  be  spent  thinking  about  the  results 

-  provide  correctly  prepared  worksheets  for  students  to  take  away  with  them 

-  provide  text  materials  ahead  of  the  class  so  students  can  read  prior  to  attending 

-  put  more  emphasis  and  examples  on  the  availability  of  data  in  the  prototype  environment 

-  address  how  findings  from  analyses  are  to  be  reported  to  management,  who  sometimes  feel 
uncomfortable  with  too  much  statistical  detail 

As  this  research  project  was  a  formative  evaluation,  feedback  from  the  Post-training  Questionnaire  and 
the  evaluators’  observations  of  the  training  sessions  was  provided  periodically  to  the  training  consultants 
so  they  could  make  adjustments  in  their  content  or  delivery. 

Implementation  of  IPPD  in  IPT  Pilots 

In  this  section,  information  from  the  small  group  and  telephone  interview  sessions  and  from  the  IPPD 
Process  Summary  Survey  are  used  in  conjunction  with  the  quantitative  responses  and  write-in  comments 
from  the  Follow-up  Questionnaire.  Results  and  attendant  discussion  generated  from  all  these  data 
collection  efforts  will  help  illustrate  the  extent  to  which  the  DSSM  and  AT/IPPD  training  participants 
were  able  to  employ  the  skills  and  knowledge  of  the  IPPD  process  and  procedures  in  their  pilot  projects. 
This  summary  of  results  will  focus  on  the  respondents’  perceptions  of 

1.  the  amount  of  IPPD  skill  and  knowledge  they  possess  and  its  sufficiency  for  applying  IPPD  in 
their  project; 

2.  the  extent  they,  individually,  and  their  pilot  teams  have  been  able  to  employ  specific  procedures 
and  tools  in  project  decisions; 

3.  the  amount  of  management  understanding  of,  and  support  for,  the  implementation  of  the  IPPD 
process  in  their  projects; 

4.  the  manner  in  which  their  projects,  to  date,  had  benefited  from  the  IPPD  training;  and 

5.  the  types  of  obstacles  they  encountered  that  impeded  IPPD  implementation. 

In  many  cases  both  overall  response  data  across  pilot  projects  and  an  indication  of  the  consistency  of 
these  responses  among  individual  pilots  are  provided.  Comparisons  of  projects  is  difficult  because  of 
variation  in  many  factors,  such  as  maturation  level  of  the  project,  degree  of  funding,  customer 
demand/support,  contractor  commitment  to  use  of  IPPD  process,  or  degree  of  conceptual  orientation  of 
the  project.  Therefore,  in  most  sections,  individual  projects  are  not  identified.  However,  a  2-3  page 
summary  of  the  IPPD  application  experience  of  each  pilot  project  evaluated  is  included  in  Appendix  F. 
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Extent  of  Present  Knowledge  and  Understanding.  Although  a  person  leaving  a  training  program  may 
have  a  certain  impression  of  the  extent  of  their  understanding  of  the  subject  matter,  that  impression  may 
change  several  months  later  when  they  have  had  to  apply  that  understanding  in  a  work  environment.  In 
the  case  of  the  IPPD  pilots,  the  perceptions  may  become  more  positive  due  to  attendance  at  additional 
supplemental  courses,  mentoring  experiences  with  training  consultants,  and  meetings  and  discussions 
associated  with  the  planning  or  actual  implementation  effort.  In  some  cases  that  perception  may  become 
less  optimistic  due  to  a  lack  of  use  of  the  information  over  an  extended  period  of  time  or  unsuccessful 
attempts  to  apply  the  process  that  make  the  IPT  members  less  confident  about  their  knowledge  and 
ability.  The  following  paragraphs  summarize  the  perceptions  that  former  participants  in  either  the 
DSSM  or  AT/IPPD  courses  had  about  their  understanding  of  the  IPPD  process  several  months  after  the 
training  experience  and  at  the  time  they  were  to  be  involved  in  implementation. 


Across  all  the  follow-up  respondents,  there  did  not  seem  to  be  a  great  deal  of  difference  from  the  end-of- 
course  responses  regarding  perceptions  of  knowledge  of  the  IPPD  concepts  and  methods  or  understand¬ 
ing  of  the  IPPD  initiative,  its  potential  benefits,  or  its  application.  As  shown  in  Table  9,  the  greatest 
difference  is  reflected  in  a  less  optimistic  perception  of  IPPD  knowledge,  with  28%  claiming  a  large  or 
very  large  amount  of  knowledge  at  least  six  months  after  training  (down  from  41%)  and  17%  claiming 
only  a  small  amount  of  knowledge  (up  from  8%).  Perceptions  of  understanding  of  the  reason  for  the 
IPPD  initiative  and  of  the  potential  benefits  were  very  similar  to  earlier  perceptions,  with  more  than  60% 
reporting  a  good  or  excellent  understanding  in  both  areas.  Perceptions  about  understanding  of  the 
application  of  IPPD  again  had  a  noteworthy  split,  with  32%  claiming  a  good  or  excellent  understanding 
and  26%  reporting  no  understanding  or  a  slight  understanding. 

Table  9.  Percentage  of  Respondents  Reporting  Ability  to  Apply  IPPD  Just  After  and  Months  After  Training 


KNOWLEDGE  OF  IPPD  CONCEPTS 

NONE  TO 

LARGE  TO  VERY 

SLIGHT 

MODERATE 

LARGE 

JUST  AFTER  TRAINING 

8% 

51% 

41% 

MONTHS  AFTER  TRAINING 

17% 

55% 

28% 

UNDERSTANDING  OF  IPPD  INITIATIVE 

NONE  TO 

GOOD  TO 

SLIGHT 

MODERATE 

EXCELLENT 

JUST  AFTER  TRAINING 

4% 

32% 

64% 

MONTHS  AFTER  TRAINING 

8% 

23% 

69% 

UNDERSTANDING  OF  IPPD  BENEFITS 

NONE  TO 

GOOD  TO 

SLIGHT 

MODERATE 

EXCELLENT 

JUST  AFTER  TRAINING 

.  5% 

34% 

60% 

MONTHS  AFTER  TRAINING 

9% 

28% 

63% 

UNDERSTANDING  OF  IPPD  APPLICATION 

NONE  TO 

GOOD  TO 

SLIGHT 

MODERATE 

EXCELLENT 

JUST  AFTER  TRAINING 

19% 

41% 

40% 

MONTHS  AFTER  TRAINING 

26% 

42% 

32% 
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Perceived  Ability  to  Apply  IPPD  to  an  Appropriate  Program.  Follow-up  perceptions  about 
individual  ability  to  apply  the  IPPD  process  are  similar  to  end-of-course  perceptions  (Table  10),  with 
28%  reporting  they  are  capable  or  very  capable  (compared  to  32%  previously)  and  34%  reporting  they 
are  slightly  or  not  at  all  capable  (compared  to  25%  previously).  The  perceived  need  for  additional 
training  before  they  would  feel  capable  of  applying  IPPD  seems  to  have  decreased  slightly.  Only  16% 
indicated  they  would  need  a  large  or  very  large  amount  of  additional  training  (compared  to  28% 
previously),  and  33%  indicated  they  would  need  no  additional  training  or  only  a  slight  amount 
(compared  to  17%  previously). 

Table  10.  Percentage  of  Respondents  Reporting  Ability  to  Apply  IPPD  Just  After  and  Months  After 
Training 

EXTENT  CAPABLE  OF  APPLYING  IPPD 
NOT  OR  SLIGHTLY  MODERATELY  CAPABLE  OR 

CAPABLE  CAPABLE  VERY  CAPABLE 

JUST  AFTER  25%  47%  28% 

TRAINING 

MONTHS  AFTER  34%  38%  28% 

TRAINING  _ _ _ 

IPPD  ADDITIONAL  TRAINING  OR  MENTORING  NEEDED 
NONE  OR  SLIGHT  AMOUNT  MODERATE  LARGE  OR  VERY  LARGE 

AMOUNT  AMOUNT 

17%  54%  28% 


33%  49%  18% 


JUST  AFTER 
TRAINING 
MONTHS  AFTER 
TRAINING 


Extent  of  Application  of  IPPD  in  Pilot  Projects.  For  this  evaluation,  several  factors  were 
analyzed  to  indicate  the  extent  the  IPPD  process  is  being  applied  as  described  in  the  AT/IPPD 
course.  These  factors  were  outlined  in  the  AT/IPPD  Training  Manual  published  by  the  James 
Gregory  training  consultants  (Brinkman  &  Peisert,  1997),  and  include: 

1 .  degree  of  customer/user  involvement  in  requirements  determination  and  throughout  the  IPT 
activities; 

2.  clear  determination  of  performance  and  technology  requirements  and  development  of  exit 
criteria; 

3 .  development  of  a  House  of  Quality  to  document  the  establishment  of  performance  and 
engineering  requirements; 

4.  analysis  of  technology/design  alternatives  based  on  the  use  of  tests,  experiments,  and/or  existent 
data  and  documented  using  Technology  Worksheets; 

5.  decisions  based  on  comparisons  of  data  relating  to  alternatives'  performance,  producibility,  cost, 
etc.,  and  documented  using  Value  Scorecards; 

6.  development  of  a  Technology  Transition  Business  Plan  (TTBP)  to  summarize  cost,  risks, 
benefits,  and  transition  assessments. 
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General  estimations  of  the  extent  to  which  the  IPT  pilots  have  employed  these  IPPD  procedures  and  that 
training  participants  have  actually  been  involved  in  this  application  are  difficult  to  achieve  only  with  data 
from  the  follow-up  questionnaire.  Differences  in  the  numbers  of  respondents  representing  various  pilot 
projects  may  make  some  overall  assessments  across  the  entire  respondent  group  misleading.  A  more 
relevant  assessment  can  be  made  using  questionnaire  results  from  each  pilot  and  a  qualitative  summary 
based  on  interviews  with  key  IPT  personnel. 

Individual  Application  of  IPPD  Procedures:  The  questionnaire  data  provide  some  indication  of  the 
extent  to  which  individual  training  participants  actually  have  been  able  to  apply  IPPD  procedures.  For 
example,  the  data  reveal  that  a  number  of  the  training  participants  claim  to  have  had  much  greater 
involvement  in  the  IPPD  application  than  others.  In  fact,  48%  of  the  respondents  indicated  they  have 
participated  "to  a  slight  extent"  or  "not  at  all"  in  the  overall  application  of  the  IPPD  process  in  their 
project.  On  the  other  hand,  25%  reported  participating  to  "a  large"  or  "very  large"  extent.  In  addition  to 
this  general  variation  in  level  of  participation  across  all  respondents,  there  is  also  a  noticeable  degree  of  variation 
among  the  different  pilots  in  terms  of  their  members’  participation  in  IPPD  application.  For  example,  Table  1 1 
shows  five  pilots  in  which  more  than  60%  of  the  respondents  report  slight  or  no  participation  in  IPPD 
application  and  three  pilots  in  which  100%  of  the  respondents  report  at  least  moderate  or  greater 
participation. 

Table  1 1 .  Percentage  of  Respondents,  by  Individual  Pilot,  Participating  at  Various  Levels  in  the 
Application  of  IPPD  to  Their  Project 


PILOT 

DID  NOT  OR 
SLIGHT  EXTENT 

MODERATE 

EXTENT 

LARGE  OR  VERY  LARGE 
EXTENT 

1 

83 

17 

0 

2 

47 

29 

24 

3 

0 

33 

67 

4 

25 

50 

25 

5 

73 

0 

27 

6 

0 

33 

67 

7 

0 

60 

40 

8 

33 

22 

44 

9 

83 

17 

0 

10 

60 

20 

20 

11 

100 

0 

0 

Tables  12-14  outlines  the  extent  to  which  the  respondents  reported  participating  in  some  of  the  individual 
IPPD  procedures.  For  example,  the  greatest  individual  involvement  seems  to  have  been  with  the  House 
of  Quality  (Table  12).  In  7  of  the  1 1  pilots,  at  least  1 7%  of  the  respondents  report  significant  or 
prominent  participation  in  the  House  of  Quality,  and  in  3  of  those  7,  at  least  60%  of  the  respondents 
report  significant  or  prominent  participation. 

Possibly  because  some  of  the  pilots  were  not  yet  at  the  stage  of  analyzing  their  design  alternatives  at  the 
time  they  filled  out  the  questionnaire,  fewer  respondents  report  participation  in  the  use  of  Technology 
Worksheets  or  Value  Scorecards.  As  shown  in  Table  13,  in  only  four  of  the  pilots  did  any  participants 
report  significant  participation  with  Technology  Worksheets. 
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Table  12.  Percentage  of  Respondents,  by  Individual  Pilot,  Participating  at  Various  Levels  in  Completing 
a  House  of  Quality  

PILOT 

DID  NOT  OR 
SLIGHT  AMOUNT 

MODERATE 

AMOUNT 

SIGNIFICANT  OR  MOST 
PROMINENT  CONTRIBUTOR 

1 

83 

0 

17 

2 

64 

18 

18 

3 

33 

0 

67 

4 

81 

6 

13 

5 

73 

0 

27 

6 

33 

0 

67 

7 

20 

20 

60 

8 

71 

0 

29 

9 

100 

0 

0 

10 

100 

0 

0 

11 

100 

0 

0 

Table  13.  Percentage  of  Respondents,  by  Individual  Pilot,  Participating  at  Various  Levels  in  Use  of 

Technology  Worksheets 

DID  NOT  OR 

MODERATE 

SIGNIFICANT  OR  MOST 

PILOT 

SLIGHT  AMOUNT 

AMOUNT 

PROMINENT  CONTRIBUTOR 

1 

100 

0 

0 

2 

94 

6 

0 

3 

100 

0 

0 

4 

73 

13 

14 

5 

91 

9 

0 

6 

33 

67 

0 

7 

75 

0 

25 

8 

67 

11 

22 

9 

100 

0 

0 

10 

60 

20 

20 

11 

100 

0 

0 

Table  14  shows  respondents  reported  even  less  participation  in  development  of  Value  Scorecards.  In 
-only  three  pilots  did  respondents  report  significant  or  prominent  participation,  and  those  respondents 
totaled  9  of  the  80  respondents.  The  pattern  that  seems  to  emerge  from  these  data  is  that  a  fairly  signifi¬ 
cant  number  of  those  who  participated  in  the  DSSM  or  AT/IPPD  training  courses  see  themselves  as 
uninvolved  in  actual  implementation  of  the  IPPD  process  in  their  pilots.  It  may  be  that,  in  many  pilots, 
the  various  IPPD  procedures/tools  are  not  used,  have  not  yet  been  used,  or  are  used  by  only  a  minority  of 
the  team  members. 

Application  of  the  IPPD  Process  Among  the  Pilot  Projects:  Of  the  13  original  IPPD  pilot  teams,  1 1 
remained  as  active  projects.  The  difficulty  in  using  questionnaire  results  to  determine  the  extent  to 
which  these  pilot  projects  are  implementing  the  IPPD  process  is  illustrated  by  the  previous  section, 
showing  the  variation  in  extent  to  which  individuals  are  involved  with  the  process.  It  is  possible  that 
some  members  are  unaware  of  the  actual  use  of  IPPD  procedures  by  other  team  members,  because  some 
team  members  report  relatively  low  levels  of  participation  and  others  report  higher  levels.  To  better 
assess  the  teams'  application  of  IPPD  procedures,  we  held  small  group  interviews  with  key  IPT 
personnel,  and  pilot  program  managers  completed  a  final  follow-up  Process  Summary  Survey.  Results 
of  this  data-gathering  indicate  there  is  considerable  variation  among  pilots  in  their  use  of  the  various 


\ 
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IPPD  procedures  presented  in  the  training  course.  Some  pilots  have  been  able  to  employ  many  of  the  six 
aspects  of  the  IPPD  process  outlined  above,  while  others  have  employed  very  few. 


Table  14.  Percentage  of  Respondents,  by  Individual  Pilot,  Participating  at  Various  Levels  in  Use  of 
Value  Scorecards 


PILOT 

DID  NOT  OR 
SLIGHT  AMOUNT 

MODERATE 

AMOUNT 

SIGNIFICANT  OR  MOST 
PROMINENT  CONTRIBUTOR 

1 

100 

0 

0 

2 

100 

0 

0 

3 

100 

0 

0 

4 

69 

19 

12 

5 

91 

9 

0 

6 

33 

33 

33 

7 

100 

0 

0 

8 

87 

13 

0 

9 

100 

0 

0 

10 

80 

0 

20 

11 

100 

0 

0 

Customer  Interaction  in  the  IPT  Activities.  As  outlined  in  the  AT/IPPD  training  program,  one  of 
the  most  basic  aspects  in  ensuring  that  affordability  is  being  considered  in  S&T  projects  is  early  and 
continuous  involvement  of  potential  users  or  customers  in  the  setting  of  requirements,  design  analysis, 
and  technology  transition  planning.  Interviews  with  pilot  team  members  indicate  that  there  was  some 
variance  in  the  extent  of  direct  customer  involvement  throughout  the  pilot  projects.  However,  most 
report  significant  involvement  in  the  early  determination  of  requirements,  especially  through  the  use  of 
meetings  specifically  dealing  with  the  technology  projects.  In  the  Summary  of  IPPD  Processes  Survey, 
program  managers  were  asked  to  select,  from  five  descriptions  of  customer  interaction,  the  description 
that  most  closely  matches  the  involvement  in  their  pilot  project.  Table  15  lists  these  options  and  the 
number  of  pilots  that  selected  each  one. 

Table  15.  Number  of  Pilots  Reporting  Various  Types  of  Customer  Interaction,  from  Choices  on 
Summary  of  Processes  Survey 

Types  of  Customer  Interaction  No.  of  Pilots  Responding 

1.  Requirements  determined  from  previous  interaction  in  research  area;  1 

little  additional  customer  involvement 

2.  Requirements  outlined/discussed  at  general  meetings  and  through  2 

documents  outlining  general  needs  (e.g.,  TPIPTs  or  deficiency  documents 

such  as  ORDS);  little  continuous  involvement  in  actual  HOQ 
development  or  other  requirements  documentation  sessions 

3.  Involved  in  determining  requirements  through  general  means  3 

described  above  and  at  initial  specific  meetings  regarding  the  technology 

project;  little  continuous  involvement  in  development  of  HOQ,  Exit 
Criteria,  or  other  requirements  documentation  sessions 

4.  Continuous  involvement  at  initial  sessions  and  in  development  of  2 

HOQ  or  other  requirements  documents 

a)  little  continuous  involvement  after  this 

b)  continuous  involvement  planned,  but  project  not  there  yet 

5.  Continuous  involvement  in  most  aspects  of  IPT,  even  after  3 

requirements  and  or  exit  criteria  were  established 


24 


-  Determination  of  Requirements.  A  second  key  aspect  of  the  IPPD  process  is  clear  determination  and 
documentation  of  requirements,  including  performance  and  engineering  requirements,  and  delineation  of 
exit  criteria.  In  a  few  pilots,  members  indicated  they  had  determined  general  requirements  from  their 
years  of  previous  experience  with  that  line  of  research  or  from  general  meetings  such  as  TPIPTs  and 
from  requirements  or  deficiency  documents;  but  they  had  developed  little  formal  documentation 
outlining  those  requirements.  However,  a  majority  of  pilots  reported  that  they  had  developed  and 
documented  formal  performance  and  engineering  requirements,  and  just  under  half  of  the  pilots  had  also 
completed  and  fully  documented  their  specific  exit  criteria.  The  Summary  of  Processes  Survey  offered 
four  potential  descriptions  regarding  the  manner  in  which  requirements  and  exit  criteria  were  developed 
and  documented.  Table  16  shows  those  choices  and  the  number  of  pilots  that  selected  each  option. 

Table  16.  Number  of  Pilots  Reporting  Various  Means  of  Determining  Requirements  and  Exit  Criteria, 

from  Choices  on  Summary  of  Processes  Survey  _ 

Means  of  Determining  Requirements  No.  of  Pilots  Responding 

1.  General  requirements  outlined  from  needs  documents  or  meetings,  but  no  3 

formal  documentation 

2.  Formal  performance  requirements  developed  and  documented,  but  no  exit  1 

criteria  documented 

3.  Formal  performance  requirements  developed  and  documented,  and  2 

technology/engineering  requirements  also  outlined  and  compared,  but  no  exit 

criteria  formally  documented 

4.  Formal  performance  requirements  developed  and  documented,  technology/  5 

engineering  requirements  outlined,  and  exit  criteria  formally  documented 


-  Use  of  a  House  of  Quality.  A  key  element  in  the  AT/IPPD  training  program  was  learning  how  to 

use  a  House  of  Quality  in  the  delineation  of  performance  and  engineering  requirements.  Many  hours 
were  spent  in  demonstrating  how  careful  use  of  this  tool  could  help  in  specification  and  prioritization  of 
requirements  and  in  early  identification  of  technology  gaps  that  could  be  costly  obstacles  in  the  design 
and  testing  of  technology  options.  While  four  of  the  pilots  indicated  that  they  did  not  use  an  HOQ,  one 
of  those  four  said  that  the  had  used  a  process  for  laying  out  requirements  that  had  a  similar  result.  Seven 
of  the  pilots  reported  that  they  were  either  working  on  or  had  completed  an  HOQ,  and  most  of  those 
suggested  that  the  process,  while  not  easy,  had  several  benefits.  The  use  of  this  tool  had  forced  them  to 
confront  requirements  issues  in  a  structured  way,  it  had  brought  constituent  members  together  to  share 
and  understand  their  differing  perspectives,  and  it  made  it  more  difficult  to  select  "favored"  technology 
options  without  having  to  meet  common  specified  criteria.  The  four  response  options  from  the  Summary 
of  Processes  Survey,  describing  the  use  of  the  HOQ,  are  listed  in  Table  17,  along  with  the  number  of 
pilots  that  indicated  their  own  experience  most  closely  matched  one  of  those  options. 

Table  17.  Number  of  Pilots  Reporting  Extent  of  Use  of  House  of  Quality,  from  Choices  on  Summary  of 


Processes  Survey _ 

Extent  of  Use  of  HOP  No.  of  Pilots 

Responding 

1.  No  House  of  Quality  developed  in  determining  requirements  3 

2.  Process  for  determining  requirements  used  that  had  similar  result  as  HOQ  (please  1 

describe  or  give  example) 

3.  HOQ  begun  and  still  working  on  it  5 

4.  HOQ  completed_ 2 
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Comparison  and  Documentation  of  Design  Alternatives  and  Completion  of  Value  Analysis.  In 
the  IPPD  process  a  systematic  approach  is  used  in  comparing  and  making  decisions  about  design  or 
technology  alternatives.  As  demonstrated  in  the  training  program,  data  are  gathered  through  testing, 
experiments,  or  other  means  to  screen  the  initial  design  options  against  customer  requirements.  For  each 
alternative  technology  considered,  performance,  producibility  and  cost  data  are  recorded  on  Technology 
Worksheets  to  identify  the  extent  of  variability  and  risk  associated  with  each  option.  All  of  these  factors 
are  compared  in  a  value  analysis  to  insure  that  process  as  well  as  product  factors  are  considered  in  the 
decision  process.  There  should  also  be  clear  documentation  of  these  factors  and  of  the  analysis  process, 
using  some  method  such  as  a  Value  Scorecard. 

Of  the  various  procedures  associated  with  the  IPPD  process,  these  two  seem  to  cause  more  difficulty  for 
the  pilots.  A  number  of  those  interviewed  contend  that  identification  of  means  for  testing  the  designs, 
actual  cost  of  testing  a  number  of  alternatives,  and  difficulty  in  obtaining  or  projecting  maintainability, 
failure  rates,  or  other  data  related  to  life  cycle  costs  has  kept  them  from  using  these  IPPD  procedures. 
They  also  contend  that  many  differences  among  design  options  are  intuitively  obvious,  and  meticulous 
documentation  of  some  of  these  considerations  is  wasteful.  Others  express  the  need  for  tools  to  help 
them  gather  the  data  for  comparisons.  Whatever  the  reason,  collection  of  data  for  testing  design 
alternatives  is  consistently  cited  as  the  most  difficult  aspect  of  IPPD  implementation.  In  spite  of  this 
difficulty,  a  majority  of  pilots  report  that  they  collect  some  data  for  comparing  design  alternatives,  and 
about  half  indicated  they  had  formally  documented  the  results;  but  only  three  reported  using  Technology 
Worksheets  or  Value  Scorecards  to  document  their  decision  process. 

In  the  Summary  of  Processes  Survey,  there  are  five  response  options  pertaining  to  the  procedures  for 
comparing  and  analyzing  design  alternatives  and  five  options  pertaining  to  procedures  for  performing  a 
value  analysis.  Tables  18  and  19  show  those  options  and  the  number  of  program  managers  who  selected 
each  one  as  being  most  closely  related  to  their  pilot's  experience. 

Table  18.  Number  of  Pilots  Reporting  Various  Means  for  Comparing  and  Documenting 

Design/Technology  Alternatives,  from  Choices  on  Summary  of  Processes  Survey 


Means  of  Comparing  Desien/Technoloev  Alternatives 

No.  of  Pilots 
Responding 

1.  Decisions  regarding  technology/design  alternatives  based  on  assessments  of 
experienced  researchers,  or  current  knowledge;  little  testing  or  data  gathering  required 

2 

2.  Decisions  regarding  technology/design  alternatives  based  on  tests,  experiments  and 
data  gathered;  little  formal  documentation  of  results 

4 

3.  Decisions  regarding  technology/design  alternatives  based  on  tests,  experiments, 
and  data  gathered;  formal  documentation  using  methods  other  than  IPPD  Technology 
Worksheets  (please  describe  method) 

2 

4.  Decisions  regarding  technology/design  alternatives  based  on  tests,  experiments, 
and  data  gathered;  formal  documentation  using  IPPD  Technology  Worksheets 

0 

5.  Project  not  at  this  point  yet,  but  intentions  are  most  likely  to  reflect  description  1 

2  3  4  above  (please  circle  ) 

3  (will  be  option  4) 
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Table  19.  Number  of  Pilots  Reporting  Various  Means  for  Carrying  Out/Documenting  a  Value  Analysis, 
from  Choices  on  Summary  of  Processes  Survey 


Means  for  Carrying  Out/Documenting  Value  Analysis 

No.  of  Pilots 
Responding 

1.  Decisions  based  on  general  intuitive  considerations  of  information  relating  to 
alternatives’  pros  and  cons;  no  formal  documentation  of  analysis  process 

1  and  1/2 

2.  Decisions  based  on  general  intuitive  considerations  of  information  relating  to 
alternatives’  pros  and  cons;  documentation  of  analysis  process  used  methods  other  than 
Value  scorecards  (please  describe  methods) 

4  and  1/2 

3.  Decisions  based  on  data  relating  to  alternatives’  performance,  producibility,  costs, 
etc.;  documentation  of  analysis  process  used  methods  other  than  Value  Scorecards 
(please  describe  methods) 

2  (1  and  2  halves) 

4.  Decisions  based  on  data  relating  to  alternatives’  performance,  producibility,  costs, 
etc. ;  documentation  of  analysis  process  included  use  of  Value  Scorecards 

0 

5.  Project  not  at  this  point  yet,  but  intentions  are  most  likely  to  reflect  description  1 

2  3  4  above  (please  circle) 

3  (will  be  option  4) 

Development  of  a  Technology  Transition  Business  Plan.  A  final  aspect  of  the  IPPD  process  that 
is  outlined  in  the  AT/IPPD  course  is  the  development  of  a  Technology  Transition  Business  plan. 
Although  most  pilots  reported  the  development  of  a  Technology  Transition  Plan  (TTP)  early  in  the 
requirements  phase  of  their  project,  none  of  them  has  yet  to  build  a  TTBP  using  the  products  of  the  IPPD 
procedures  demonstrated  in  the  course.  One  program  manager  indicated  his  pilot  had  developed  a 
TTBP-like  document  that  summarized  their  assessments  of  cost,  risk,  benefits,  and  transition  factors 
without  using  the  HOQ  and  IPPD  design  analysis  tools,  and  three  others  indicated  they  intend  to 
complete  a  TTBP  as  their  project  matures.  There  are  four  response  options  pertaining  to  the 
development  of  a  TTBP  in  the  Summary  of  Processes  Survey;  Table  20  shows  the  number  of  pilots  that 
associate  their  TTBP  experience  with  one  of  those  options. 

Table  20.  Number  of  Pilots  Reporting  Extent  of  Development  of  a  Technology  Transition  Business  Plan 
(TTBP),  from  Choices  on  Summary  of  Processes  Survey 


Extent  of  Development  of  a  TTBP 

1.  TTBP  not  developed 

2.  TTBP  developed  summarizing  cost,  risk,  benefits,  and  transition  assessments, 
arrived  at  through  Value  Scorecard,  HOQ  requirements  and  other  IPPD  products 

3.  TTBP  developed  summarizing  cost,  risk,  benefits,  and  transition  assessments, 
arrived  at  by  means  other  than  IPPD  products 

4.  Project  not  at  this  point  yet,  but  intentions  are  most  likely  to  reflect  description  1 
2  3  above  (please  circle) 


No.  of  Pilots 
Responding 

6 

0 

1 

4  (will  be  option  2) 
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Participation  in  Supplemental  Training  and  Mentoring  Opportunities 

To  aid  in  educating  IPPD  pilot  team  members  and  assisting  them  with  IPPD  implementation  in  their 
projects,  training  consultants  provided  three  supplemental  courses  to  the  AT/IPPD  or  DSSM  participants. 
They  presented  a  Design  of  Experiments  (DOE)  course,  a  Design  To  Cost  (DTC)  course,  and  a  short 
DSSM  course.  In  addition,  four  members  who  had  taken  the  original  DSSM  course  in  1996  also 
attended  an  AT/IPPD  course  in  1997.  Consultants  also  were  available  for  individual  mentoring  visits  to, 
and  communications  with,  the  various  pilots  and  offered  to  participate  in  the  pilots’  IPT  sessions. 

As  of  November  1997,  the  DOE  course  was  best  attended,  with  63  members  from  pilot  IPTs  attending. 
Thirteen  IPT  members  attended  the  Design  to  Cost  course,  and  four  former  AT/IPPD  attendees  also 
attended  a  two-day  DSSM  course.  Of  all  those  taking  supplemental  training,  12  IPT  members  took  two 
additional  courses  and  three  took  three.  From  comments  made  during  the  interview  sessions  and  on  the 
Follow-up  Questionnaire,  the  DOE  course  was  considered  very  beneficial.  Some  members  indicated  it 
ought  to  be  a  required  course  for  all  those  entering  the  laboratory  R&D  environment.  Table  21  shows  the 
number  and  percentage  of  respondents  who  report  attending  supplemental  courses.  It  reveals  that  two 
thirds  of  the  respondents  took  advantage  of  the  additional  educational  opportunity,  having  attended  at 
least  one  additional  course,  and  22%  attended  two  or  more. 

Table  21.  Respondents'  Participation  in  IPPD  Supplemental  Courses _ 


NUMBER  OF  ADDITIONAL 

PERCENTAGE  OF 

COURSES 

NUMBER  OF  RESPONDENTS 

RESPONDENTS 

0 

28 

33% 

1 

39 

45% 

2 

16 

19% 

3 

3 

3% 

TOTAL 

86 

100% 

Mentoring  or  ongoing  technical  support  to  IPTs,  both  before  and  after  training,  included  IPPD  process 
implementation  planning  sessions,  IPT  review  attendance,  and  assistance  with  requirements  deter¬ 
mination,  HOQ  development,  design  of  experiments,  and  development  of  tools  or  methods  to  measure 
performance  requirements  or  other  factors  considered  in  analysis  of  design  options.  Respondents  to  the 
Follow-up  Questionnaire  indicated  engaging  in  as  many  as  20  additional  mentoring  interactions,  some  of 
which  could  have  included  telephone  guidance  and  assistance.  Table  22  shows  the  number  of  mentoring 
communications  reported  by  the  Follow-up  respondents.  It  is  interesting  that  55%  of  the  respondents 
report  no  mentoring  contact,  while  10%  report  at  least  six  or  more  mentoring  communications. 

The  researchers  attempted  to  determine  if  there  was  any  indication  of  a  relationship  between  involvement 
with  supplemental  courses  or  mentoring  activities  and  perceptions  of  knowledge,  understanding,  and 
ability  to  apply  IPPD  procedures.  To  check  this  relationship,  they  compared  the  responses  of  those  who 
had  attended  different  numbers  of  courses,  and  they  also  compared  the  responses  of  those  who  reported 
differing  numbers  of  mentoring  contacts.  Across  all  responses  dealing  with  knowledge,  understanding, 
and  perceived  ability  to  apply,  there  appeared  to  be  no  meaningful  differences  between  those  who  had 
taken  no  additional  courses  and  those  who  had  taken  one  or  two  additional  courses.  There  was  a 
consistently  higher  perception  of  knowledge,  understanding,  and  ability  to  apply  IPPD  among  those  who 
had  taken  three  additional  courses,  but  the  small  number  of  respondents  in  that  category  prevents  any 
conclusion  of  a  relationship  between  number  of  courses  and  perceived  levels  of  IPPD  knowledge, 
understanding,  and  ability  to  apply  the  process. 
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Table  22.  Number  of  Mentoring  Incidents  Reported  by  Respondents 


NUMBER  OF  MENTORING 

PERCENTAGE  OF 

INCIDENTS 

NUMBER  OF  RESPONDENTS 

RESPONDENTS 

0 

55 

65% 

1 

6 

7% 

2 

4 

5% 

3 

7 

8% 

4 

4 

5% 

6 

5 

6% 

8 

1 

1% 

10 

1 

1% 

16 

1 

1% 

20 

1 

1% 

TOTAL 

85 

100% 

This  same  type  of  comparison  was  made  among  those  who  reported  differing  numbers  of  mentoring 
communications.  Respondents  were  grouped  by  those  who  had  no  mentoring  contacts  (n  =  54),  those 
who  reported  between  one  and  five  contacts  (n  =  21),  and  those  who  reported  six  or  more  (n  =  9).  Table 
23  shows  the  comparison  of  responses  regarding  knowledge  of  the  IPPD  process.  It  is  evident  that  those 
with  six  or  more  contacts  feel  most  optimistic  about  their  IPPD  knowledge,  while  those  with  a  few 
mentoring  contacts  report  the  least  amount  of  knowledge.  Table  24  shows  the  same  pattern  of 
differences  among  the  respondents’  perceptions  of  their  ability  to  apply  the  IPPD  process.  In  fact,  this 
pattern  is  consistent  across  all  of  the  knowledge,  understanding,  and  application  ability  responses. 


Table  23.  Amount  of  IPPD  Knowledge  Reported  by  Respondents  with  Various  Amounts  of  Mentoring 
_ Communication _ _ _ 

PERCENT  REPORTING  AMOUNTS  OF  IPPD  KNOWLEDGE 


NUMBER  OF 

MENTORING  LARGE  TO 

INCIDENTS  SLIGHT  OR  NONE  MODERATE  VERY  LARGE 


0 

19% 

1-4 

19% 

6  + 

0% 

54% 

28% 

62% 

19% 

44% 

56% 

Table  24.  Ability  to  Apply  IPPD,  Reported  by  Respondents  with  Various  Amounts  of  Mentoring 
Communication 

PERCENT  REPORTING  CAPABILITY  OF  APPLYING  IPPD  PROCESS 
NUMBER  OF 

MENTORING  LARGE  TO 

INCIDENTS  SLIGHT  OR  NONE  MODERATE  VERY  LARGE 

0  35%  38%  27% 

1-4  38%  43%  19% 

6+  11%  33%  56% 


29 


Additional  support  and  a  probable  explanation  for  this  relationship  between  a  larger  amount  of  mentoring 
and  greater  confidence  in  IPPD  knowledge  and  ability  is  provided  by  the  reported  level  of  individual 
involvement  with  IPPD  application  in  their  respective  pilots.  As  seen  in  Table  25,  the  heavily  mentored 
group  all  report  having  participated  to  a  large  or  very  large  degree  in  their  IPPD  application,  while  those 
receiving  less  mentoring  also  report  far  less  participation.  It  is  likely  that  the  heavily  mentored 
respondents  are  all  key  players  in  their  projects’  IPPD  efforts,  and  they  have  sought  a  lot  of  additional 
guidance  so  they  could  become  proficient  in  that  role. 

Table  25.  Extent  of  Participation  in  Application  of  IPPD  to  Project,  Reported  by  Respondents  with 
Various  Amounts  of  Mentoring  Communication 


PERCENT  REPORTING  EXTENT  OF  PARTICIPATION  IN  IPPD  APPLICATION 


NUMBER  OF 
MENTORING 
INCIDENTS 

SLIGHT  OR  NONE 

MODERATE 

LARGE  TO  VERY 
LARGE 

0 

58% 

29% 

13% 

1-4 

38% 

33% 

29% 

6  + 

0 

0 

100% 

The  point  that  may  be  considered  here  is  one  of  immersion:  that,  when  introducing  a  new  process  or 
concept,  after  a  certain  level  of  involvement  and  guidance,  confidence  in  one’s  ability  is  finally  gained, 
and  a  level  of  comfort  with  the  new  process  is  reached.  The  more  important  relationship  that  may 
develop  once  this  point  is  reached  is  commitment  to  or  belief  in  the  new  process.  As  shown  in  Table  26, 
almost  all  (89%)  of  those  who  reported  a  great  deal  of  mentoring  (and  also  who  were  all  actively 
involved  in  their  IPPD  application)  also  report  a  great  likelihood  that  they  will  employ  IPPD  in  future 
projects.  Although  there  may  be  many  other  explanations  for  this  relationship,  this  view  is  also 
supported  by  the  comments  of  several  of  those  interviewed  who  expressed  a  positive  view  of  the 
potential  for  implementing  IPPD  in  their  projects.  Those  most  positive  all  referred  to  past  experiences 
(sometimes  in  a  manufacturing  environment)  in  which  they  had  successfully  employed  IPPD  (or  very 
similar)  procedures.  They  commented  that  it  might  take  a  lot  of  effort  to  convince  their  less  committed 
colleagues  that  the  IPPD  process  will  be  beneficial  over  “business  as  usual,”  but  their  experience  level 
with  the  process  has  kept  them  committed  to  that  effort. 

Table  26.  Likelihood  of  Applying  IPPD  in  Future  Projects,  Reported  by  Respondents  with  Various 
Amounts  of  Mentoring  Communication 

PERCENT  REPORTING  LIKELIHOOD  OF  APPLYING  IPPD  IN  FUTURE 
NUMBER  OF 

MENTORING  UNLIKELY  OR  LIKELY  OR 

INCIDENTS  HIGHLY  UNLIKELY  50%  CHANCE  HIGHLY  LIKELY 

0  25%  40%  35% 

1-4  24%  38%  38% 

_ 6+ _  0  11%  89% 
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Perceptions  of  Management’s  Knowledge/Understanding/Attitude/Support 

A  key  factor  in  the  implementation  of  any  new  program  requiring  personnel  to  change  their  methods  of 
operating  is  the  perception  of  management  commitment  and  support.  In  fact,  the  importance  of  this 
factor  was  recognized  by  the  DOD  S&T  Affordability  Task  Force  when  they  issued  their  “Criteria  for 
Assessing  a  Focus  on  Affordability  in  S&T  Program  Management”  in  1997.  One  of  the  five  major  areas 
outlined  as  essential  for  evaluating  the  affordability  initiative  is  management’s  willingness  and  ability  to 
support  Integrated  Product  and  Process  Development.  Both  through  the  Follow-up  Questionnaire  and  the 
interviews,  researchers  gathered  information  reflecting  the  pilot  members’  perceptions  of  management’s 
understanding  of  IPPD  and  belief  in  the  ability  to  apply  IPPD  successfully  in  the  S&T  environment. 

Data  pertaining  to  those  perceptions  were  previously  gathered  from  participants  at  the  AT/IPPD  training 
sessions  and  are  reported  above  (Figures  9  -  14).  The  follow-up  questions  were  designed  to  capture  the 
perceptions  of  IPPD  pilot  members  as  they,  and  their  managers,  actually  faced  the  difficulties  of 
implementing  new  procedures  within  the  constraints  of  the  workplace  environment. 

After  six  or  seven  more  months  of  experience  with  the  prospect  of  applying  IPPD  in  their  workplace,  the 
respondents  did  not  have  any  more  optimistic  view  than  they  had  going  through  training,  of 
management’s  understanding  of  IPPD  or  of  their  belief  that  IPPD  is  being  successfully  applied.  In  their 
estimates  of  management’s  knowledge  of  the  process  and  understanding  of  how  to  apply  it  to  their 
project  (summarized  in  Table  27),  at  least  50  %  of  the  respondents  reported  either  not  knowing 
management’s  degree  of  understanding  or  feeling  it  is  slight  or  nonexistent. 

Table  27.  Percentage  of  Respondents  Reporting  Management's  Levels  of  IPPD  Knowledge  and 
Understanding 

KNOWLEDGE  OF  IPPD  CONCEPTS  AND  METHODS 

MODERATE  LARGE  OR 

DON’T  KNOW  NONE  OR  SMALL  AMOUNT  AMOUNT  VERY  LARGE  AMOUNT 

7%  49%  29%  15% 


UNDERSTANDING  OF  HOW  IPPD  METHODS  APPLY  TO  PROJECT 
DON’T  KNOW  NONE  OR  SLIGHT  MODERATE  GOOD  OR  EXCELLENT 

9%  51%  24%  15%  


The  respondents’  perceptions  are  similar  when  it  comes  to  management’s  opinion  of  how  successfully 
IPPD  has  so  far  been  applied  and  the  amount  of  benefit  that  IPPD  has  brought  to  the  project.  As  shown 
in  Table  28,  at  least  67%  either  didn’t  know  management’s  opinion  or  felt  management  thought 
application  was  weak  or  nonexistent.  The  table  also  shows  that  nearly  80%  either  didn’t  know 
management’s  opinion  or  felt  management  thought  IPPD  had  brought  small  or  no  benefits. 


Table  28.  Percentage  of  Respondents  Reporting  Management's  Perceptions  about  Success  of  IPPD 

Application  in  Pilot  Projects  and  the  Extent  of  Benefits  Gained _ 

LEVELS  OF  SUCCESS  OF  IPPD  APPLICATION 

WEAK  OR  SUCCESSFUL  OR 

DON’T  KNOW  NONEXISTENT  MEDIOCRE  HIGHLY  SUCCESSFUL 

28%  39%  19%  14% 

EXTENT  OF  BENEFITS  FROM  IPPD 

DON’T  KNOW  NONE  OR  SMALL  MODERATE  LARGE  OR  VERY  LARGE 

30%  49%  16%  5% 
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Beyond  the  perceptions  of  management's  understanding  of  IPPD  and  of  management's  opinions  of  its 
application  and  benefit,  are  the  respondents'  perceptions  of  the  support  for  those  steps  needed  for 
successful  implementation.  Support  for  attendance  at  initial  IPPD  training  is  perceived  to  have  been 
relatively  strong.  Table  29  shows  67%  of  respondents  reporting  "strong"  or  "very  strong"  management 
support  for  this  training.  Support  for  supplemental  training  or  mentoring,  however,  was  reported  to  be 
much  less  enthusiastic.  Only  28%  of  respondents  reported  receiving  strong  or  very  strong  support  for 
these  additional  means  of  assistance,  and  36%  indicated  their  support  was  slight  or  nonexistent. 

Probably  most  important  were  perceptions  of  management  support  for  the  actual  use  of  IPPD  methods  in 
the  project.  Again,  only  3 1%  of  respondents  reported  strong  or  very  strong  support,  while  8%  did  not 
know  what  level  of  support  they  had,  and  33%  felt  their  support  was  slight  or  nonexistent.  These  data, 
also  displayed  in  Table  29,  when  combined  with  the  perceptions  of  management  knowledge  and  attitudes 
expressed  at  the  AT/IPPD  training  sessions,  do  not  reflect  that  a  message  of  strong  management  support, 
deemed  essential  by  the  S&T  Affordability  Task  Force,  is  being  received  by  the  pilot  team  members. 


Table  29.  Percentage  of  Respondents  Reporting  Extent  of  Management  Support  for  IPPD  Training 
and  Use  of  IPPD  Methods 


MANAGEMENT  SUPPORT  FOR  INITIAL  IPPD  TRAINING 

DON’T  KNOW 

NONE  OR  SLIGHT 

MODERATE 

STRONG  OR 

VERY  STRONG 

3% 

4% 

26% 

67% 

MANAGEMENT  SUPPORT  FOR  SUPPLEMENTAL  TRAINING  OR  MENTORING  IJSF, 

DON’T  KNOW 

NONE  OR  SLIGHT 

MODERATE 

STRONG  OR 

VERY  STRONG 

11% 

36% 

26% 

28% 

MANAGEMENT  SUPPORT  FOR  USE  OF  IPPD  METHODS 

DON’T  KNOW 

NONE  OR  SLIGHT 

MODERATE 

STRONG  OR 

VERY  STRONG 

8% 

33% 

28% 

31% 

Perceptions  of  Benefits  to  Projects 

One  of  the  most  difficult  things  to  ascertain  in  this  evaluation  is  whether  the  IPPD  training  and  resultant 
implementation  provided  benefits  to  the  pilot  projects.  To  help  answer  this  question,  researchers 
gathered  information  both  from  the  Follow-up  Questionnaire  and  the  group  interviews.  The  Question¬ 
naire  respondents  were  asked  to  estimate,  generally,  the  extent  their  program  had  benefited  from  use  of 
IPPD  methods  and,  specifically,  the  extent  to  which  the  team's  application  of  IPPD  had  helped  reduce 
costs,  shorten  timelines,  increase  productivity,  or  identify  risks.  In  addition,  they  were  asked  about  the 
usefulness  of  the  IPPD  course  and  how  important  or  beneficial  it  would  be  for  others  to  take  the  course. 

Early  in  the  series  of  interview  sessions,  the  research  team  realized  the  determination  of  benefits  is 
confounded  by  several  factors.  These  include  differences  in  the  extent  to  which  various  pilots  actually 
implemented  the  IPPD  process,  the  lack  of  knowledge  of  many  respondents  about  their  team's 
implementation,  and  the  fact  that  several  of  the  pilots  had  not  progressed  far  enough  along  to  really  allow 
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for  noticeable  benefits  to  be  realized.  Although  the  questionnaire  data  in  general  revealed  little  in  the 
way  of  tangible  benefit  to  the  areas  described,  the  team  members  interviewed  did  describe  some  less 
concrete  ways  in  which  the  IPPD  effort  helped  their  programs.  In  the  following  paragraphs  we  will 
provide,  first,  a  brief  summary  of  the  respondents'  perceptions  of  benefits  described  in  the  Questionnaire, 
and  then  a  short  summary  of  the  benefits  described  by  those  interviewed. 

Perceived  Benefits  of  Application  of  IPPD  Methods.  When  asked  to  estimate  the  extent  to  which  the 
use  of  IPPD  methods  benefited  their  pilot  project,  relatively  few  respondents  reported  large  or  even 
moderate  benefits.  As  shown  in  Table  30,  27%  indicated  IPPD  benefited  the  overall  pilot  project 
moderately,  and  only  7%  reported  IPPD  benefited  the  project  “greatly”  or  “excellently.”  That  pattern 
holds  when  asked  about  specific  types  of  benefits,  such  as  helping  reduce  costs,  increase  productivity, 
shorten  timelines,  or  identify  risks.  Table  3 1  shows  that  a  fairly  large  percentage  (as  much  as  38%)  of 
respondents  indicated  they  don’t  know  whether  IPPD  use  benefited  any  of  these  areas,  and  over  50% 
consistently  indicated  benefits  to  these  areas  were  slight  or  nonexistent.  The  most  benefit  was  reported 
to  be  in  the  identification  of  risks,  where  33%  indicated  IPPD  benefited  this  area  at  least  moderately  or 
greater.  An  interesting  side  note  to  these  perceptions  is  that  respondents  did  see  greater  potential  benefit 
from  implementation  of  IPPD  methods~in  other  6.3  programs.  As  seen  in  Table  30, 46%  indicated  IPPD 
would  benefit  other  programs  to  a  large  or  very  large  extent,  and  86%  said  it  would  benefit  other 
programs  to  a  moderate  extent  or  greater.  This  may  be  an  indication  that  the  general  logic  and 
philosophy  of  the  IPPD  process  is  appreciated,  but  that  the  teams  were  not  able  to  implement  them  with 
much  noticeable  benefit  in  these  first  pilots  by  the  time  the  Questionnaire  data  were  collected. 

Table  30.  Respondents’  Perceptions  of  Extent  IPPD  Application  Provided  or  Would  Provide  Overall 

Benefits  to  Their  Project  or  Other  Projects  _ 

PERCENT  REPORTING  EXTENT  IPPD  USE  BENEFITED  THEIR  PROGRAM 

NONE  OR  SLIGHTLY  MODERATELY  GREATLY  OR  EXCELLENTLY 

_ 66% _ 27% _ 7% _ 

PERCENT  REPORTING  EXTENT  IPPD  COULD  BENEFIT  OTHER  6.3  PROGRAMS 
NONE  OR  SMALL  MODERATE  LARGE  OR  VERY  LARGE 

14%  40%  46% 


Table  31.  Respondents’  Perceptions  of  Extent  IPPD  Application  Benefited  Specific  Aspects  of  Their 

Project _ _ _ 

pERCENT  reporting  IPPD  HELPED  REDUCE  COSTS 
NOT  AT  ALL 

DON’T  KNOW  OR  SLIGHT  MODERATE  LARGE  OR  VERY  LARGE 

38%  53%  6%  3% 

PERCENT  REPORTING  EPPD  HELPED  SHORTEN  TIMELINES 
NOT  AT  ALL 

DON’T  KNOW  OR  SLIGHT  MODERATE  LARGE  OR  VERY  LARGE 

34%  58%  8%  0 

PERCENT  REPORTING  IPPD  HELPED  INCREASE  PRODUCTIVITY 
NOT  AT  ALL 

DON’T  KNOW  OR  SLIGHT  MODERATE  LARGE  OR  VERY  LARGE 

22%  58%  16%  4% 

PERCENT  REPORTING  IPPDi  HELPED  IDENTIFY  RISKS 
NOT  AT  ALL 

DON’T  KNOW  OR  SLIGHT  MODERATE  LARGE  OR  VERY  LARGE 

15%  52%  27%  6% 
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Respondents’  Perceptions  of  Usefulness  of  the  IPPD  Course.  Although  respondents  did  not  report  a 
great  amount  of  benefit  to  their  programs  (yet)  from  the  actual  implementation  of  IPPD  methods,  most 
did  seem  positive  about  the  utility  of  the  IPPD  course.  Table  32  shows  that  46  %  indicated  that  the 
information  provided  would  be  moderately  useful  for  their  general  professional  development,  and 
another  42%  reported  it  would  be  quite  or  extremely  useful.  In  addition,  the  respondents  indicated  that  it 
was  important  for  other  members  of  their  team  to  take  the  course.  Greater  than  75%  of  the  respondents 
reported  that  it  was  important  for  all  government  members  and  all  contractor  members  of  the  team  to 
take  the  course,  either  at  the  same  time  or  at  some  point.  The  response  regarding  the  utility  of  the  IPPD 
course  for  colleagues  doing  similar  work,  but  who  are  not  part  of  the  IPPD  pilot  project,  is  most  telling. 
As  shown  in  Table  32,  38%  indicated  it  was  appropriate  for  some  of  their  colleagues,  and  24%  indicated 
that  it  was  more  than  just  appropriate  or  that  they  would  strongly  recommend  it  for  those  colleagues. 


Table  32.  Respondents'  Perceptions  of  Training  Utility 


PERCENT  REPORTING  IPPD  COURSE  USEFUL  FOR  PROFESSIONAL  DEVELOPMENT 

NO  USE  OR 

MODERATELY 

OUITE  OR  EXTREMELY 

VERY  LITTLE 

USEFUL 

USEFUL 

12% 

46% 

42% 

PERCENT  REPORTING  IMPORTANCE  OF  ALL  GOVERNMENT  MEMBERS  ATTENDING  COURSE 

NOT  IMPORTANT 

YES.  AT  SOME  POINT 

YES.  AT  SAME  TIME 

23% 

39% 

38% 

PERCENT  REPORTING  IMPORTANCE  OF  ALL  CONTRACTORS  ATTENDING  COURSE 

NOT  IMPORTANT 

YES.  AT  SOME  POINT 

YES.  AT  SAME  TIME 

22% 

37% 

41% 

PERCENT  RECOMMENDING  IPPD  COURSE  FOR  COLLEAGUES  OR  OTHER  PROEJCTS 


NO 

WILL  PROVIDE  A 
LITTLE  BENEFIT 

APPROPRIATE  FOR  SOME 

STRONGLY 

RECOMMENDED 

9% 

28% 

38% 

24% 

Perceptions  of  Ancillary  Benefits  of  IPPD  Implementation.  Despite  the  relatively  small  numbers  of 
Questionnaire  respondents  who  reported  seeing  substantial  benefits  to  specific  areas  of  their  pilot 
projects,  those  members  interviewed  repeatedly  mentioned  ways  in  which  their  attempt  to  implement 
IPPD  procedures  had  assisted  their  projects.  The  most  frequently  mentioned  benefit  was  seen  in  their 
efforts  to  specify  performance  and  engineering  requirements  with  a  House  of  Quality.  Members  from  at 
least  five  of  the  pilots  indicated  that  this  practice  had  helped  (or  forced)  them  to  use  a  structured  means 
for  articulating  performance  and  engineering  factors  that  might  have  been  given  less  emphasis  in  the 
past.  It  gave  everyone  a  common  way  of  defining  and  prioritizing  the  requirements  and  forced  them  to 
confront  how  they  would  measure  those  requirements.  It  also  made  it  more  difficult  for  any  group  to 
lobby  for  a  specific  process  unless  they  could  show  how  it  stacked  up  against  the  stated  requirements. 
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The  second  most  frequently  cited  benefit  was  the  emphasis  on  increased  communication  with  users  and 
customers  in  the  definition  of  requirements  and  exit  criteria.  Members  of  some  projects  said  competing 
customer  requirements  sometimes  made  the  development  of  a  House  of  Quality  difficult  and,  in  some 
cases,  the  customers  grew  impatient  with  the  structured  HOQ  process.  However,  the  consensus  was  that 
bringing  in  the  potential  users  aided  in  requirements  definition,  gave  them  a  greater  stake  in  the  entire 
R&D  process,  and  made  it  more  possible  to  obtain  needed  data  (such  as  for  cost  or  supportability 
comparisons). 

Another  benefit  cited  was  the  positive  impact  on  team  interaction  and  communication  that  IPPD 
implementation  brought  about.  Not  only  with  customers,  but  also  with  various  contractor  groups  and 
even  among  different  government  stakeholders,  the  need  to  gather  structured  input  for  such  procedures 
as  the  HOQ  or  to  determine  how  the  team  was  going  to  implement  the  IPPD  procedures  forced  greater 
interaction  and  sharing  of  different  perspectives.  The  setting  up  of  Web  sites  and  more  frequent 
teleconferences  enhanced  communication  among  different  groups,  and  more  structured  documentation  of 
requirements  and  decision  parameters  provided  a  better  trail  of  the  decision  processes  and  rationale  and 
lessened  the  opportunity  for  misunderstandings. 

The  pilot  team  members  who  described  these  benefits  indicated  that  these  are  all  factors  that  should  be 
taken  into  account  in  any  effective  R&D  team  effort.  However,  the  rigor  of  the  IPPD  process  lessened 
the  tendency  to  let  some  of  these  practices  slide.  In  fact,  for  some  of  the  pilots,  there  was  somewhat  of  a 
"Hawthorne  effect."  That  is,  the  IPPD  process'  emphasis  on  more  rigorous  procedures  for  defining 
requirements  and  testing  options  caused  even  members  of  some  of  those  pilots  that  were  less  able  to 
implement  IPPD  to  reexamine  the  ways  they  made  decisions  and  consider  how  they  were  addressing 
affordability. 


Obstacles  to  Implementation  of  IPPD 

As  they  described  their  experiences  in  implementing  IPPD  methods  in  their  pilot  projects,  team  members 
identified  several  factors  that  made  it  difficult  to  apply  the  process  outlined  in  the  training  program.  In 
the  interview  sessions,  the  IPT  members'  comments  seemed  to  coalesce  around  three  main  areas:  lack  of 
sufficient  tools  or  methods  to  accomplish  the  robust  testing  and  data-gathering  needed  in  the  design 
analysis  and  value  analysis  phases  of  the  IPPD  process;  lack  of  time  and  resources,  within  their  current 
project  funding,  to  develop  the  tools  and  accomplish  the  additional  testing,  data  gathering,  and 
documentation  required;  and  an  inability  to  adapt  the  IPPD  process  to  the  particular  nature  of  the  project 
because  of  its  very  early  conceptual  stage  or  its  software  engineering  orientation. 

In  some  of  the  projects,  the  IPT  members  indicated  that  it  would  be  hard  to  gather  data  on  some  of  the 
design  factors  they  would  be  considering  because  (a)  there  were  few  or  no  existing  data  on  these  new 
concepts,  (b)  they  didn't  have  the  means  to  build  and  carry  out  repeated  tests  cost-effectively,  and  (c) 
they  had  not  developed  simulations  or  other  surrogate  means  for  gathering  those  data.  Therefore  they 
indicated  they  collected  harder  data  where  it  actually  existed,  carried  out  tests  where  they  had  the  tools  to 
do  so  cost  effectively,  and  relied  on  intuitive  analyses  for  other  comparisons.  Some  of  the  program 
managers  also  indicated  that  they  had  been  able  to  develop  some  metrics  and  means  for  experimental 
testing  with  the  help  of  the  training  consultants.  They  stressed  that  additional  help  with  developing  tools 
for  R&D  data  analysis  was  necessary  for  IPPD  to  be  more  widely  applied. 
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Several  programs  suggested  that  development  of  tools  and  carrying  out  tests  might  have  been  possible, 
but  the  actual  costs  and  time  required  to  do  so  would  not  fit  into  their  funding  parameters.  Those  costs 
had  not  been  part  of  the  original  budget  and  contractors  were  not  willing  to  absorb  the  additional  costs. 
Interviewees  indicated  that  requirements  for  completing  rigorous  testing  would  have  to  be  written  into 
contracts  and  accompanied  by  additional  funding  needed  to  develop  the  methods  and  carry  out  the  tests. 

The  third  commonly  perceived  obstacle  was  that  the  very  early  conceptual  nature  of  the  project  meant 
that  many  of  the  conditions  and  variables  were  changing  as  new  knowledge  led  to  new  approaches  and 
requirements.  One  manager  indicated  the  IPPD  procedures  were  much  more  appropriate  for  a  later  phase 
of  his  project,  when  the  earliest  concepts  had  been  worked  out  and  requirements  and  design  parameters 
would  be  less  fluid. 

Related  to  these  concerns  were  the  perceptions  of  IPT  members  from  two  software  engineering  projects 
who  indicated  that  they  had  not  been  able  to  determine  how  to  apply  the  IPPD  procedures  to  an  iterative 
design  process  used  in  software  engineering.  The  training  consultants  will  be  working  with  members  of 
one  of  the  software  projects  to  see  how  the  IPPD  procedures  could  be  used. 

In  addition  to  comments  at  the  interviews,  information  related  to  perceptions  of  these  obstacles  was 
gathered  through  responses  to  the  Follow-up  Questionnaire.  In  the  earlier  Participant  Questionnaire, 
administered  at  the  training  sessions,  respondents  had  been  asked  to  estimate  the  influence  that  four 
factors  might  have  in  preventing  IPPD  application  (see  Table  8).  In  the  Follow-up  Questionnaire, 
respondents  were  asked  to  use  the  same  scale  to  indicate  the  amount  the  same  four  factors,  plus  an 
additional  three  factors  identified  through  discussions  with  pilot  members,  had  played  in  hindering  the 
successful  application  of  IPPD  in  their  pilot  projects.  Results  from  the  latter  instrument  seem  to  support 
some  of  the  perceptions  expressed  in  the  interviews. 

Table  33  shows  respondents'  perceptions  regarding  the  impact  of  these  various  obstacles.  It  is  evident 
that  the  most  prominent  obstacle  was  one  of  those  just  cited  above:  that  there  are  still  too  many  unknown 
or  changing  variables  in  the  early  stage  of  the  project  to  apply  IPPD.  In  seven  of  the  pilots,  at  least  50% 
or  more  of  the  respondents  indicated  this  factor  hindered  the  IPPD  application  a  large  or  very  large 
amount.  The  problem  of  the  project's  special  needs  was  the  next  frequently  cited  obstacle,  with  at  least 
50%  of  the  respondents  in  five  pilots  indicating  it  hindered  to  a  large  or  very  large  degree.  The  third 
most  cited  obstacle  was  the  lack  of  resources  and  materials.  The  fourth  most  frequently  cited  obstacle 
was  "other,"  which  actually  included  a  cross  section  of  different  write-in  factors,  including  more  detailed 
explanations  of  the  seven  listed  factors.  The  most  prominent  of  the  different  “other”  factors  were  (a)  the 
pilot  projects  were  at  too  immature  a  stage  to  benefit  from  the  IPPD  process,  (b)  not  having  sufficient 
members  of  the  IPT  trained  in  IPPD  procedures,  and  (c)  geographic  separation  of  team  members. 


CONCLUSIONS  AND  RECOMMENDATIONS 


Based  on  the  results  from  all  of  the  information-gathering  efforts  described  above,  the  researchers  offer 
the  following  conclusions  regarding  the  initial  AT/IPPD  and  DSSM  training  programs  and  the  pilots’ 
implementation  of  IPPD  procedures.  They  also  include  several  recommendations  that  may  make  future 
training  and  implementation  more  effective. 
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Table  33.  Percent,  by  Pilot,  Indicating  Various  Factors  Hindered  Their  IPPD  Implementation  a  Large 
or  Very  Large  Amount _ 


PILOT  TEAMS 


Hindering  Factors 

1 

2 

3 

4 

5 

6 

7 

8 

2 

10 

11 

Too  many  unknown  or 

100 

25 

67 

60 

55 

33 

80 

0 

83 

60 

0 

changing  variables 

Project  has  special  needs/ 

33 

19 

67 

53 

18 

33 

40 

0 

100 

80 

75 

qualities  not  to  IPPD 

Too  far  along  in  project  to 

0 

13 

100 

20 

0 

0 

0 

0 

83 

60 

50 

employ 

Not  enough  time 

17 

44 

0 

40 

45 

0 

0 

0 

17 

60 

25 

Not  enough  knowledge  or 

33 

12 

67 

20 

9 

0 

0 

11 

0 

0 

0 

training  on  IPPD 

Not  enough  resource 

33 

31 

33 

20 

54 

0 

60 

11 

17 

20 

100 

materials  available 

Not  enough  support  or 

33 

10 

67 

13 

18 

0 

0 

11 

0 

0 

50 

emphasis  from 
management 

Other 

33 

25 

67 

33 

26 

100 

20 

11 

50 

20 

25 

The  Initial  AT/IPPD  Training  Session 

The  AT/EPPD  Course  Initially  Improved  the  Knowledge,  Understanding,  and  Attitudes  of 
Attendees.  Based  on  feedback  at  the  training  sessions  and  from  the  Participant  Questionnaire,  it  is 
evident  that  the  AT/IPPD  training  participants  perceived  themselves  as  having  greater  knowledge  and 
understanding  of  the  IPPD  process  than  they  had  prior  to  training.  They  also  reported  more  positive 
perceptions  about  the  likelihood  that  the  process  could  be  applied  successfully  to  their  project  and  that  it 
was  likely  to  provide  actual  benefits.  This  course  was  designed  as  an  introduction  to  most  of  the  major 
processes,  and  was  not  intended  to  enable  the  participants  to  apply  the  procedures  covered  without 
further  training  or  assistance.  That  limited  objective  seemed  to  be  borne  out  by  the  comments  and 
responses  of  the  learners,  who  had  absorbed  a  lot  of  information  but  knew  they  would  need  much  more 
knowledge  and  skill  than  they  had  acquired  in  the  class  in  order  to  implement  IPPD  in  their  projects. 


Changes  Should  Be  Made  to  Make  the  AT/EPPD  Course  Even  More  Effective.  Despite  the  reported 
cognitive  and  affective  gains,  the  initial  AT/IPPD  course  was  in  need  of  some  improvement  to  make  it 
more  acceptable  and  relevant  for  the  S&T  audience.  Remarks  during  the  course  and  write-in  comments 
on  the  Questionnaire  revealed  several  areas  of  concern.  Most  common  was  the  desire  to  have  the  course, 
especially  the  URV  exercise,  more  related  to  R&D  situations,  and  preferably  to  situations  encountered  by 
the  pilot  projects.  There  was  a  strong  feeling  that  too  much  material  was  attempted  in  the  time  period, 
and  that  segments  were  too  math-intensive  for  a  portion  of  the  learners.  It  was  obvious  from  observing 
the  exercises  that  there  was  a  fairly  large  difference  in  statistical  capability  among  the  training 
participants  and  that  some  trainees  were  lost  and  frustrated  at  times.  It  was  also  obvious  from  one  of  the 
first  sessions  that  holding  the  training  at  the  work  site  led  to  participants'  attempting  to  deal  with  work 
issues  in  between  (and  even  during)  class  modules.  Off-site  training  was  accomplished  for  most  of  the 
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other  sessions.  This  and  other  feedback  were  provided  to  the  training  consultants,  and  they  attempted  to 
make  some  adjustments  in  pace  and  in  the  exercise  by  their  later  sessions.  The  consultants  seemed 
sensitive  to  continuously  improving  the  effectiveness  of  the  IPPD  education  process,  and  they  have 
indicated  they  are  developing  a  new  version  of  the  AT/IPPD  course  to  address  the  concerns  identified  in 
the  first  round  of  pilots. 

Supplemental  Courses  and  Mentoring  Assistance  Were  Necessary  for  Application  to  Projects.  As 
discussed  above,  the  trainees  generally  came  away  from  the  AT/IPPD  course  feeling  somewhat  positive 
about  the  IPPD  process  and  willing  to  try  to  apply  it,  but  knowing  that  they  would  need  additional  help 
or  training  before  they  could  implement  the  process  in  their  projects.  This  had  been  anticipated  by  the 
consultants,  and  supplemental  training,  in  the  form  of  the  Design  of  Experiments,  Design  to  Cost,  and 
DSSM  courses,  were  available  and  recommended  by  the  training  team.  The  DOE  course  was  heavily 
attended  and  was  well  received  by  those  who  took  it.  Numerous  comments  during  interviews  indicated 
that  the  DOE  trainees  found  it  applicable  to  their  work  and  thought  it  should  be  a  standard  course  for 
S&T  laboratory  researchers.  Fewer  people  took  the  other  two  courses,  and  there  was  less  feedback  about 
their  applicability.  For  some  who  took  the  DSSM  course  the  orientation  was  still  too  manufacturing- 
related.  Comparisons  of  data  from  the  Follow-up  Questionnaire  regarding  perceived  IPPD  knowledge 
and  understanding  among  those  who  took  different  numbers  of  supplemental  courses  showed  little 
difference  between  those  who  took  no  additional  courses  and  those  who  took  one  or  two  courses. 

More  important  than  the  supplemental  courses  were  the  additional  mentoring  opportunities  provided  by 
the  training  consultants  to  individual  pilots.  Most  groups  who  made  any  progress  with  the  application  of 
IPPD  indicated  they  relied  on  the  assistance  of  the  consultants  to  help  them  outline  requirements,  build 
an  HOQ,  or  even  determine  metrics  or  tools  for  gathering  data  for  decision  analyses.  Comparisons 
regarding  levels  of  knowledge  and  understanding  among  those  who  had  different  amounts  of  mentoring 
contact  supported  the  efficacy  of  that  assistance.  Those  who  had  received  six  or  more  mentoring 
contacts  indicated  they  were  much  more  involved  with  the  application  of  IPPD  in  their  projects  and  that 
they  had  a  relatively  greater  level  of  knowledge  and  understanding  of  the  IPPD  process.  More 
important,  they  also  indicated  that  they  were  very  likely  to  use  the  IPPD  procedures  in  other  projects. 
This  was  most  likely  a  group  of  key  players  in  their  project's  IPPD  implementation,  who  were  working 
closely  with  the  consultants,  trying  to  make  the  process  work.  This  information  indicates  the  education 
process  for  effective  implementation  of  IPPD  is  ongoing  and  requires  assistance  of  knowledgeable 
consultants  through  the  first  implementation  experiences.  It  also  is  likely  that,  when  that  assistance  is 
received  and  the  implementers  reach  a  level  of  comfort  with  the  process,  their  belief  in  its  value  and 
motivation  to  continue  its  use  increases.  Thus,  for  the  long-term  implementation  of  IPPD,  the  goal 
should  be  to  get  as  many  of  the  researchers  as  possible  to  that  comfort  level. 

Potential  Exists  for  Employing  Alternative  Means  of  Training  Delivery.  As  they  observed  the 
AT/IPPD  sessions  and  discussed  the  types  of  post-training  interaction  that  took  place  between  project 
teams  and  the  consultants,  the  researchers  were  also  considering  the  potential  for  alternative  means  of 
training  delivery.  Their  conclusion  is  that  the  introductoiy  AT/IPPD  course  ought  to  be  taught  in  the 
traditional  classroom  mode,  but  that  there  are  ample  opportunities  for  exportable  training  in  the  follow¬ 
up  activities  and  assistance  that  will  help  the  implementers  as  they  apply  the  process.  The  AT/IPPD 
course  is  an  introductory  learning  experience  for  those  who  are  about  to  engage  in  the  IPPD  application 
process.  From  the  audiences  observed  in  the  original  training  sessions,  many  come  to  this  experience 
with  a  great  deal  of  skepticism  about  this  new  process  that,  they  believe,  demands  a  change  in  the  way 
they  traditionally  have  been  operating.  Introduction  of  a  new  process  frequently  requires  an  attitudinal 
change  as  well  as  development  of  additional  knowledge  and  skills.  In  this  environment  it  is  essential  for 
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the  facilitator  to  continually  read  the  learners'  attitude  and  motivation,  sense  and  deal  with  frustration, 
and  apply  personal  attention  when  necessary.  In  addition,  the  interaction  with  and  facilitation  of  the 
teams  during  the  exercises  enables  group  members  to  learn  from  one  another  in  those  areas  with  which 
they  feel  comfortable.  With  these  requirements,  exportable  interactive  courseware  or  internet-based 
training  could  not  match  the  instructors'  ability  to  derive  immediate,  often  nonverbal,  signals  and  respond 
with  an  alternate  example  or  explanation  that  will  ease  the  fears  of  the  learners  and  enhance  their 
motivation. 

However,  there  seems  to  be  a  lot  of  opportunity  for  alternative  delivery  of  training  after  the  introductory 
course,  for  remedial  or  tutorial  purposes.  Modules  containing  tutorial  explanation  and  exercises  can  be 
accessed  from  a  web  site  and  reviewed  by  pilot  team  leaders  or  members  just  prior  to  the  attempted 
application  of  a  particular  procedure  (e.g.,  development  of  an  HOQ  session).  Internet  or  even 
teleconference  communication  can  be  used  during  IPT  sessions  to  monitor  progress  and  provide  advice 
when  the  group  reaches  a  difficult  point.  As  the  team  members  get  more  confident  in  implementing  the 
procedures  and  become  more  positive  about  the  process,  motivation  may  become  less  of  a  factor  in  the 
learning  process  and  less  direct  hands-on  interaction  will  be  necessary. 

Implementation  of  IPPD  in  the  Pilot  Programs 

The  IPPD  Process  Seems  to  Be  Working  in  Several  Projects,  but  Implementation  Has  Been 
Inconsistent  across  Procedures  and  Pilots.  At  this  point,  the  IPPD  process  is  clearly  not  a  simple 
formula  that  can  easily  be  applied  uniformly  to  all  6.3  S&T  projects.  The  implementation  of  IPPD  in  the 
S&T  environment,  like  the  IPPD  training,  is  a  work  in  progress.  The  consultants  as  well  as  the  various 
programs  are  learning  how  to  apply  the  process  to  the  variety  of  situations  encountered  in  the  R&D 
world.  Some  aspects  seem  to  be  more  easily  applied  than  others.  For  example,  the  emphasis  on  early 
customer  involvement  and  on  determination  of  performance  and  engineering  requirements  and 
development  of  specific  exit  criteria  have  been  accomplished  by  most  of  the  pilot  teams.  In  most 
instances,  the  IPT  members  have  indicated  that  attempts  to  use  these  procedures  have  had  a  positive,  but 
not  readily  quantifiable,  effect  on  their  project.  Some  pilot  team  members  have  questioned  whether  the 
amount  of  extra  effort  required  to  involve  the  customers  more  fully,  lay  out  the  requirements  more 
specifically,  and  formally  document  them  with  a  House  of  Quality  are  worth  it  in  the  long  run,  especially 
when  their  funding  and  timelines  are  tight.  Ironically,  for  a  group  of  researchers,  they  need  to  be 
reminded  that  the  first  time  through  a  new  process  usually  requires  much  more  time  and  effort  than  will 
be  needed  when  the  implementers  have  more  experience  with  its  application. 

Other  IPPD  aspects,  especially  those  related  to  data  gathering,  testing,  and  documentation  for  the 
comparison  of  design/technology  alternatives  and  for  the  value  analysis,  have  been  less  rigorously  or 
consistently  applied  in  the  early  pilots.  It  is  in  these  areas  that  additional  progress  must  be  made  in  the 
education  of  the  S&T  community.  Tools  or  methods  for  data  gathering  must  be  developed,  and  in-depth 
training  or  facilitation  must  be  provided  to  enable  project  members  to  accomplish  the  requisite 
comparisons  and  analyses  needed  for  making  affordability  decisions.  At  this  point,  the  data-gathering 
issue,  whether  it  is  lack  of  knowledge,  methods,  or  motivation  to  put  forth  the  additional  effort,  seems  to 
be  a  major  sticking  point  in  the  consistent  application  of  the  IPPD  process. 

What  does  seem  evident  at  this  time  is  that  some  of  the  pilot  projects  are  making  more  progress  in  IPPD 
implementation  and  seem  more  comfortable  with  the  procedures  than  other  pilots.  There  appear  to  be 
four  pilots  whose  efforts  more  closely  reflect  those  recommended  in  the  AT/IPPD  course.  The  NGT 
project  has  aggressively  embraced  the  IPPD  procedures  from  the  onset,  and  the  CAI  Pervasive  project, 
ALEP,  and  IRCM  all  have  been  actively  working  at  implementation.  Of  the  remaining  seven  pilots, 


39 


three  seem  to  have  been  able  to  employ  some  of  the  recommended  procedures  but  are  struggling  with 
others;  three  (3-DOC,  OTH,  and  WINDS)  showed  basically  very  little  application  of  the  IPPD  process; 
and  one  (ISCP)  did  little  initially  but  seems  much  more  involved  with  implementation  in  a  second  phase. 

It  is  indicative  of  the  evolving  nature  of  this  IPPD  learning  process,  that,  as  this  report  is  being  put  in 
final  form,  the  OTH  pilot  representatives  have  met  with  the  training  consultants  to  develop  metrics  for 
evaluating  relocatable  over-the-horizon  radar  technologies  to  be  introduced  into  their  enhanced 
demonstration  system.  If  they  indeed  do  develop  and  apply  these  affordability  metrics,  they  will  be 
employing  some  of  the  IPPD  procedures  that  they,  heretofore,  had  felt  were  not  applicable.  This 
example  illustrates  both  the  need  for  follow-on  interaction  with  the  consultants  and  the  difficulty  of 
evaluating  IPPD  implementation  without  continuous  updating  of  program  status. 

Original  Selection  of  Pilots  Could  Have  Been  Better.  One  of  the  factors  that  has  affected  the 
inconsistent  application  of  IPPD  across  the  first  group  of  pilots  is  the  fact  that  some  of  the  projects 
selected  were  not  optimally  suited  to  be  part  of  this  test  group.  Two  of  the  pilots  selected  were  quite  far 
along  in  their  work  by  the  time  they  had  taken  the  IPPD  or  DSSM  training.  One  of  those  actually  went 
back  and  recreated  some  of  the  earlier  decision  processes  using  the  procedures  recommended  by  the 
IPPD  process.  The  other  of  those  two  projects  simply  felt  too  far  along  and  did  not  implement  the  IPPD 
procedures.  For  other  pilots,  the  implementation  was  constrained  because  the  requirements  for  IPPD 
procedures  were  not  written  into  the  contracts  and  the  contractors  balked  at  having  to  use  more  rigorous 
or  different  methods  than  they  had  originally  intended.  For  another,  the  program  manager  was  not 
aware  at  the  start  of  the  project  of  the  additional  funding  required  to  do  the  rigorous  testing.  And  no 
additional  funding  was  made  available,  so  fewer  of  the  IPPD  procedures  were  accomplished.  Another 
project  actually  lost  some  of  its  funding  during  the  time  it  was  involved  as  a  pilot. 

Several  Factors  Have  Been  Cited  as  Necessary  for  Supporting  the  Training  Efforts  and  Increasing 
the  Likelihood  of  Successful  Implementation.  All  the  discussions  with  members  of  pilot  teams, 
consultants,  and  IPPD  implementation  managers  have  provided  insight  about  some  factors  that  seem 
related  to  more  successful  implementation  efforts.  The  following  are  the  most  prominent  considerations 
mentioned  consistently: 

Upper  Management  Commitment  to  Establishment  of  the  IPPD  Process  Must  be  Strong  and  Visible. 
Both  Air  Force  and  industry  experience  have  shown  that,  when  a  large  change  in  philosophy  or  operating 
procedures  is  introduced,  one  of  the  reactions  is  for  the  workforce  to  wait  and  see  whether  the  initiative 
is  going  to  survive  before  they  make  a  strong  effort  to  embrace  it.  Experience  with  several  Air  Force 
programs  (e.g.,  Zero  Defects,  Management  By  Objectives,  and  more  recently,  the  Total  Quality  Initiative 
-  TQI)  has  made  many  Air  Force  personnel  suspicious  about  the  lasting  effect  of  such  initiatives.  One  of 
the  signs  that  the  initiative  is  a  serious  effort  that  is  not  going  to  just  go  away  is  the  extent  to  which 
management  shows  enthusiastic  support  for  and  confidence  in  the  change  effort.  This  requires  them  to 
be  knowledgeable  about  the  new  program  and  willing  to  spend  the  time  and  resources  to  effect  the 
change.  The  data  from  both  questionnaires  as  well  as  the  comments  of  trainees  and  interview  groups, 
indicate  many  of  the  pilot  members  are  not  getting  a  strong  signal  that  management  is  truly  supportive  of 
or  even  knowledgeable  about  this  IPPD  process. 


Contractor  Commitment  to  IPPD  Must  Be  Obtained  and  Maintained Several  program  managers 
expressed  their  difficulty  in  getting  contractors  to  cooperate  with  the  IPPD  procedures  because  it 
required  them  to  adopt  new  methods  or  do  additional  work.  Contractors  also  stated  that  their  original 
bids  were  based  on  standard  operating  procedures,  and  that  requirements  for  newer  procedures,  that  take 
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additional  effort,  must  be  written  into  the  contract.  Those  pilots  that  were  able  to  do  this  or  who  had 
contractors  who  were  more  familiar  with  IPPD  procedures  indicated  their  implementation  task  was  much 
easier. 

Follow-up  Education  and  Involvement  With  Mentoring  is  Necessary.  Most  of  the  pilots  that  seem 
more  successful  with  implementation  of  IPPD  have  cited  strong  involvement  with  training  consultants 
after  the  initial  training  session.  The  data  also  show  those  most  inclined  to  use  IPPD  in  other  projects  are 
those  who  have  worked  most  closely  with  mentors  and  have  reached  a  fairly  strong  level  of  knowledge 
and  understanding. 

Funding  the  Additional  Effort  Associated  with  IPPD  is  Necessary.  Both  those  who  have  had 
difficulty  implementing  IPPD  and  those  who  have  been  more  successful  indicate  that  obtaining  funding 
needed  for  that  effort  is  essential.  As  mentioned  above,  having  to  take  the  additional  money  for 
additional  IPPD  training  or  mentoring  or  implementation  of  IPPD  procedures  has  been  a  stopper  for 
some  programs.  Two  of  the  programs  experiencing  more  success  have  been  able  to  shift  money  within 
their  budgets  to  cover  the  additional  costs.  It  is  much  less  likely  to  have  a  program  implement  IPPD 
when  its  funding  is  being  constantly  under  fire  or  reduced. 

Program  Manager/Leadership  Should  Be  Knowledgeable  About  and  Committed  to  the  Process. 

What  seems  quite  clear  is  that  the  projects  showing  most  progress  with  IPPD  implementation  have 
program  managers  or  key  team  leaders  that  are  somewhat  familiar  with  the  process  or  a  similar  way  of 
doing  business.  They,  because  of  their  experience  or  belief  in  the  process,  seem  more  willing  to  work 
with  the  consultants  and  struggle  through  the  learning  process,  developing  the  tools  and  methods  needed 
to  apply  the  process.  They  admit  to  being  willing  to  encourage  their  contractors’  or  other  team  members’ 
participation  and  less  willing  to  accept  the  members’  position  that  “this  is  too  hard,  the  process  doesn’t 
work.” 

It  Is  Difficult  to  Determine  Whether  There  Are  Differences  in  Applicability  by  Type  of  Project  or 
Laboratory.  Because  there  were  so  many  additional  variables  impacting  the  implementation  of  the 
IPPD  process  among  the  different  pilots,  the  question  of  whether  the  process  is  equally  applicable  across 
all  types  of  6.3  S&T  projects  or  at  different  laboratories  remains  difficult  to  answer.  The  timing  of  the 
training  and  the  introduction  of  the  process,  the  funds  and  time  available  for  working  with  consultants  to 
develop  tools,  the  willingness  of  contractors  to  implement  IPPD  procedures,  and  the  general  motivation 
of  IPT  members  varied  across  the  pilots.  Plus,  the  small  number  of  pilots  with  similar  conditions  makes 
generalization  very  risky.  It  does  seem  clear  that  the  application  of  the  process  in  software  engineering 
projects  has  yet  to  be  worked  out.  This  does  not  mean  that  some  aspects  of  the  process  would  not  be 
helpful  to  these  projects  or  that  better  means  of  applying  the  process  could  not  be  developed.  It  simply 
reflects  the  fact  that,  in  this  round,  the  two  pilots  that  were  primarily  software  engineering  projects  were 
unable  to  determine  how  to  employ  the  process.  The  fact  that  one  of  the  projects  was  very  far  along 
when  it  began  its  IPPD  affiliation  further  confounded  that  effort. 

The  situation  is  the  same  across  laboratories.  There  were  two  projects  each  at  each  of  three  laboratories 
and  six  at  another.  One  lab  had  two  projects,  which  did  not  employ  IPPD.  But  one  of  those  projects  was 
the  software  project  that  started  IPPD  very  late  in  the  life  of  the  project.  A  second  lab  is  involved  with 
two  projects,  and  one  of  those  is  a  software  project.  The  third  lab  involved  with  two  projects  had  one 
canceled  as  an  IPPD  pilot  due  to  contractor  difficulties.  In  essence,  the  small  numbers  of  projects 
remaining  that  are  not  confounded  by  "special  circumstances"  makes  it  impossible  to  generalize  about 
the  applicability  of  IPPD  among  various  labs. 
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The  IPPD  Training  Evaluation  Effort 


The  Present  Evaluation  Effort  Had  Some  Limitations,  But  Gathered  Relevant  Data.  As  described 
in  the  Methodology  section  of  this  report,  the  researchers  employed  several  means  for  evaluating  the 
AT/IPPD  training  program  and  the  trainees'  ability  to  implement  IPPD  procedures  outlined  in  that 
program  in  their  pilot  projects.  In  the  determination  of  the  AT/IPPD  course  and  the  immediate  effect  on 
the  skills,  knowledge  and  attitudes  of  the  training  participants,  the  following  methods  were  employed: 

1 .  observation  of  and  participation  in  each  of  the  AT/IPPD  training  sessions  to  experience  what  the 
trainees  were  going  through,  the  relative  ease  or  difficulty  of  attaining  the  desired  objectives,  and  the 
methods  and  pace  of  the  presentation; 

2.  discussions  with  trainees  and  trainers  during  the  course  to  get  a  sense  of  their  level  of  motivation 
toward  the  subject  matter,  and  their  own  perceptions  regarding  high  and  low  points  of  the  training 
program;  and 

3.  a  Participant  Questionnaire  to  capture  perceptions  about  pre-  and  post-training  levels  of 
knowledge,  understanding,  ability  to  apply  the  process,  and  attitudes  regarding  the  likelihood  of  IPPD 
being  successfully  applied  and  benefiting  the  projects. 

To  assess  the  extent  to  which  the  trainees  were  able  to  implement  the  procedures  taught  in  the  course  and 
the  confounding  factors  beside  skills  and  knowledge  that  affect  that  IPPD  application,  the  researchers 
used  a  similar  variety  of  methods: 

1 .  small  group  and  telephone  interviews  with  a  sampling  of  government  and  contractor  members  of 
the  various  pilot  teams; 

2.  a  Follow-up  Questionnaire  administered  to  training  participants  who  were  still  associated  with  the 
pilot  teams;  and 

3.  a  final  Summary  of  Processes  Survey,  asking  program  managers  to  select,  from  statements 
describing  different  levels  of  involvement  with  each  of  six  aspects  of  the  IPPD  process,  the  descriptions 
that  most  closely  match  their  own  pilot's  status. 

Although  additional  methods  were  considered,  some  limitations  in  timing  and  access  to  the  pilot  team 
members  prevented  their  use  in  this  initial  evaluation.  For  example,  the  researchers  were  not  totally 
pleased  with  relying  on  self-report  perceptions  of  levels  of  IPPD  skill  and  knowledge.  Ideally,  they 
would  have  liked  to  develop  and  administer  an  objective  test  to  measure  mastery  of  the  cognitive 
learning  objectives.  In  addition,  because  each  project  was  different  and  had  so  many  different 
confounding  circumstances,  it  was  difficult  to  tell  whether  different  implementation  outcomes  were 
related  to  the  training  or  to  other  conditions  associated  with  the  project.  And,  as  with  most 
questionnaires,  even  after  initial  pretesting,  the  researchers  would  still  have  altered  some  of  the  questions 
and  scales.  In  general,  however,  the  data  and  information  gathered  were  able  to  provide  a  fairly  balanced 
picture  of  the  immediate  effects  of  the  training  course  and  of  the  level  of  implementation  of  the  IPPD 
procedures  in  the  various  projects. 


42 


Recommendations  for  a  Long-term,  Follow-on  Evaluation  .  Based  on  the  initial  experience  with 
methods  and  instruments  in  this  environment,  the  researchers  offer  several  recommendations  for  a  long¬ 
term  evaluation  effort. 

1.  Multiple  methods  of  assessment  across  multiple  information  sources  are  necessary.  While 
structured,  scaled  responses  to  questionnaire  items  are  useful  for  quantifying  information  across  larger 
numbers  of  subjects,  comments  captured  from  open-ended  questionnaire  items,  and  especially  from 
discussions  and  interviews,  give  better  indications  of  attitude,  motivation,  frustrations,  and  other 
reactions  that  might  be  hidden  in  the  responses  to  simple  scaled  items.  In  addition  it  is  necessary  to 
interview  and  gather  questionnaire  information  from  a  cross-section  of  the  pilot  team  members. 
Contractors,  customers,  and  other  government  members  as  well  as  the  program  managers  will  give  a 
more  balanced  picture  and  help  determine  if  the  program  manager  is  candid  or  providing  a  politically 
acceptable  picture. 

2.  Because  so  many  of  the  criteria  for  assessing  the  implementation  of  the  IPPD  procedures  (e.g.,  the 
extent  of  customer/user  involvement)  are  not  easily  quantified,  evaluators  will  have  to  continue  to  rely 
on  subjective  assessments.  These  assessments  should  be  refined  as  the  training  consultants  and  the  IPTs 
gain  more  experience  with  the  implementation  of  IPPD  in  the  S&T  environment  and  develop  more 
realistic  expectations  about  the  outcomes  that  can  result  from  application  of  those  procedures.  Presently 
those  expectations  are  still  being  discovered. 

3.  Evaluation  teams  should  be  present  at  the  beginning  of  each  project's  IPPD  planning  and 
implementation,  prior  to  training. 

4.  Evaluators  should  attend  the  training  sessions  to  determine  the  participants’  degree  of 
commitment  to  the  training,  identify  any  problems  with  a  specific  training  experience,  and  see  the 
consultants  are  attempting  improvements. 

5.  Members  of  the  evaluation  team  should  be  familiar  with  the  laboratory  S&T  environment  and  the 
acquisition  process.  This  helps  to  determine  the  extent  to  which  difficulties  in  implementation  are  due  to 
lack  of  skill  or  knowledge,  lack  of  tools  or  metrics  to  gather  data,  lack  of  commitment  to  implementation 
on  the  part  of  some  team  members,  or  other  confounding  conditions,  such  as  timing  or  funding. 

6.  Ideally,  a  member  of  the  evaluation  team  should  have  experience  with  successful  implementation 
of  the  IPPD  process.  A  leader  from  one  of  the  current  successful  pilots  would  be  a  credible  addition. 

7.  Ideally,  evaluators  would  have  access  to  nonproprietary  documents  from  IPT  review  sessions  and 
project  status  reports. 

8.  Assessment  should  be  periodic  and  continuous  to  determine  the  changes  in  implementation  status 
and  to  identify  the  effects  of  additional  training  or  mentoring  efforts. 

General  Recommendations 

The  researchers  realize  that  implementation  of  IPPD  in  the  S&T  environment  is  a  work  in  progress. 

From  the  beginning  of  the  training,  the  IPPD  managers,  training  consultants,  and  the  evaluation  team 
were  aware  that  the  initial  AT/IPPD  course  was  a  necessary  introduction  to  the  process,  but  was  far  from 
sufficient  to  instill  the  skills  and  motivation  needed  to  change  the  work  behavior  of  experienced 
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professionals.  This  formative  evaluation  project  is  one  method  of  reviewing  this  “first  effort”  and 
identifying  ways  to  improve  both  the  training  and  the  structure  for  supporting  it.  In  that  spirit,  the 
following  recommendations  are  provided: 

1 .  Select  a  new  series  of  pilots  based  on  criteria  that  will  foster  success.  The  next  group  should 
avoid  the  obstacles  of  program  timing,  funding,  and  contractor  reluctance  that  hindered  implementation 
in  the  first  series  of  pilots. 

2.  In  the  projects  selected,  discretionary  funding  ought  to  be  planned  for  additional  training, 
mentoring,  metrics  development,  and  testing  that  are  likely  to  be  encountered  with  IPPD  implementation. 
Program  managers  will  need  to  know  how  estimate  these  costs. 

3.  Use  of  IPPD  methodology  should  be  included  in  the  Statement  of  Work  and  other  contracting 
documents  to  be  sure  contractors  are  aware  of  those  requirements.  Experience  with  the  IPPD  process 
ought  to  be  one  of  the  criteria  for  consideration  of  contract  awarding.  Program  managers  will  have  to 
know  how  to  word  these  documents  and  estimate  IPPD  experience 

4.  The  program  managers  and  key  leadership  ought  to  be  knowledgeable  about  the  IPPD  process 
and/or  committed  to  making  it  work.  And,  as  indicated  in  the  conclusions  above,  the  program  leadership 
has  to  be  strong  enough  to  enforce  that  commitment  throughout  their  IPTs. 

5.  Additional  IPPD  training  should  be  developed  for  senior  management  and  program  managers. 

For  senior  managers  the  general  philosophy  and  processes  should  be  outlined  without  requiring  the 
extensive  mathematical  detail  of  the  AT/IPPD  course.  More  important  is  the  discussion  of  standard 
obstacles  that  may  impede  implementation  and  methods  to  prevent  or  circumvent  these  and  ensure 
motivation  among  the  IPTs.  For  program  managers  the  same  sort  of  discussion  of  obstacles  ought  to  be 
accompanied  by  some  instruction  and/or  guidelines  for  estimating  costs  of  IPPD  implementation  and  for 
ensuring  commitment  to  IPPD  methods  among  the  contractors.  These  short  courses  ought  to  include 
participation  by  experienced  IPPD  implementers  from  the  laboratories  who  have  successfully  used  the 
process  in  their  S&T  projects  and  who  can  share  their  stories  and  describe  what  it  took  to  overcome  the 
nay-sayers. 

6.  Most  important,  the  IPPD  education  initiative  must  be  viewed  as  a  long-term  incremental  process. 
Changing  long-held  practices  is  not  accomplished  with  a  one-shot  training  effort.  Mentoring  and  follow¬ 
up  with  individual  pilots  is  essential  to  develop  the  tools  and  methods  needed  for  the  S&T  environment 
and  to  establish  a  small  group  of  success  stories.  The  goal  ought  to  be  to  select  carefully  the  next  group 
of  pilots  and  provide  intensive  support  so  that  many  more  members  get  to  that  comfort  level  which  leads 
to  belief  in  the  process.  Then,  from  that  group,  the  S&T  community  will  have  a  small  core  of 
experienced  facilitators  within  the  various  labs.  That  group  will  have  credibility  because  their  peers 
know  them  and  yet  will  have  conquered  the  same  fears  and  potential  obstacles  that  the  next  group  of 
pilots  is  facing.  The  goal  of  that  core  of  facilitators  is  to  bring  about  more  success  stories  and  more 
believers  who  achieve  that  comfort  level.  It  is  this  incremental  approach,  rather  than  an  attempt  at  a 
broad-based,  surface-level  inoculation  of  training  that  will  have  a  lasting  effect  on  the  way  the  S&T 
community  approaches  affordability. 
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APPENDIX  A:  PARTICIPANT  QUESTIONNAIRE 
Evaluation  Questionnaire 


Name _ j _ 

Organization  _ 

Telephone _ 

email  _ 

Instructions:  The  purpose  of  this  questionnaire  is  to  gather  information  that  will  be  used  to 
evaluate  and  refine  not  just  this  course,  but  the  IPPD  initiative  in  the  science  and 
technology  environment  in  general.  As  you  may  know,  pilot  team  members  and  other 
interested  parties  will  be  asked  to  supply  Information  via  survey,  telephone  interview,  or 
other  means  from  time  to  time  during  the  evaluation  program.  A  full  evaluation  effort  will 
require  extensive  information  about  how  and  how  well  the  IPPD  process  is  implemented 
across  pilot  projects.  We  therefore  request  your  name  and  other  information  so  that  we 
can  follow  up  on  this  questionnaire  if  necessary.  However,  we  will  not  provide  or  report 
identifiable  individual  responses  to  anyone.  Whenever  an  item  refers  to  “your  6.3  project,” 
assume  that  it  refers  to  the  pilot  6.3  project  to  which  the  IPPD  process  is  being  applied  and 
for  which  you  are  now  being  trained. 


1.  Are  you  either  working  on  or  in  some  substantial  way  connected  with  one  of  the  IPPD  6.3  pilot 
projects? 

Yes _  No _  (If  your  answer  is  “no”  please  go  to  item  la, 

after  which  you  are  finished  with  this  survey.) 

la.  Answer  only  if  you  answered  “no”  to  question  1 :  Since  you  are  not  working  on  or  in  some  substantial 
way  connected  with  one  of  the  IPPD  6.3  pilot  projects,  please  explain  briefly  why  you  were  in  the  course. 


2.  Which  response  best  describes  your  employment  category? 


Government 

Contractor 

Other  Please  explain 

(Military  or 

briefly  below  (e.g.,  if  you  consider 

civil  service) 

yourself  a  customer  or  user  for 
a  6.3  program,  please  say  so.) 

3.  Which  of  the  following  best  describes  your  current  position  (circle  one): 

(a)  Technical  Manager/Supervisor 

(b)  Administrative  Manager/Supervisor 

(c)  Research/Bench  Level  Engineer/Scientist 

(d)  Contracting  Officer 

(e)  Other  (please  describe  briefly): 
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4.  How  long  have  you  been  in  your  current  position?  _  years  _ months 

5.  How  long  have  you  been  working  in  your  career  field?  _ years  _ months 

6.  On  a  scale  of  1  -  5,  below,  indicate  how  much  prior  experience  you  have  had  working  as  a  member  of 
an  Integrated  Product  Team  (IPT)? 

1  2  3  4  5 

no  small  amount  moderate  amount  large  amount  very  large  amount 

experience  of  experience 

7.  Indicate  how  much  you  knew  about  IPPD  concepts  and  methods  prior  to  attending  this  course. 

1  2  3  4  5 

no  small  amount  moderate  amount  large  amount  very  large  amount 

knowledge  of  knowledge 

8.  To  what  extent  did  you  understand  -  prior  to  attending  this  course  -  the  reasons  for  the  IPPD  initiative? 

1  2  3  4  5 

no  slight  moderate  good  excellent 

understanding  understanding  understanding  understanding  understanding 

9.  To  what  extent  did  you  understand  -  prior  to  attending,  this  course  -  the  intended  benefits  of  the  IPPD 
initiative? 

1  2  3  4  5- 

no  slight  moderate  good  excellent 

understanding  understanding  understanding  understanding  understanding 

10.  To  what  extent  did  you  understand  -  prior  to  attending  this  course  -  how  IPPD  methods  apply  to  your 
6.3  program? 

1  2  3  4  5 

no  slight  moderate  good  excellent 

understanding  understanding  understanding  understanding  understanding 

11-  Prior  to  attending,  this  course .  how  likely  did  you  think  it  would  be  that  the  IPPD  initiative  could  be 
successfully  applied  to  your  6.3  program? 

0  1  2  3  4  5 

had  no  application  application  chances  application  application 

opinion  highly  unlikely  unlikely  about  50%  likely  highly  likely 

12.  Prior  to  attending  this  course .  how  likely  did  you  think  it  would  be  that  IPPD  methods  could  be 
applied  to  your  6.3  project  with  real  benefits  to  the  program? 

0  1  2  3  4  5 

had  no  real  benefits  real  benefits  chances  real  benefits  real  benefits 

opinion  highly  unlikely  unlikely  about  50%  likely  highly  likely 


* 
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13.  Indicate  the  extent  of  your  present  knowledge  about  IPPD  concepts  and  methods. 


1  2  3  4  5 

no  small  amount  moderate  amount  large  amount  very  large  amount 

knowledge  of  knowledge 

14.  To  what  extent  do  you  presently  understand  the  reasons  for  the  IPPD  initiative? 

1  2  3  4  5 

no  slight  moderate  good  excellent 

understanding  understanding  understanding  understanding  understanding 

15.  To  what  extent  do  you  presently  understand  the  intended  benefits  of  the  IPPD  initiative  ? 

1  2  3  4  5 

no  slight  moderate  good  excellent 

understanding  understanding  understanding  understanding  understanding 

16.  To  what  extent  do  you  presently  understand  how  IPPD  methods  apply  to  your  6.3  program? 

1  2  3  4  5 

no  slight  moderate  good  excellent 

understanding  understanding  understanding  understanding  understanding 

17.  Presently,  how  likely  do  you  think  it  is  that  the  IPPD  initiative  can  be  applied  successfully  to  your  6.3 
program? 

1  2  3  4  5 

application  application  chances  application  application 

highly  unlikely  unlikely  about  50  %  likely  highly  likely 

18.  Presently,  how  likely  do  you  think  it  is  that  IPPD  methods  can  be  applied  to  your  6.3  project  with  real 
benefits  to  the  program? 

1  2  3  4  5 

real  benefits  real  benefits  chances  real  benefits  real  benefits 

highly  unlikely  unlikely  about  50%  likely  highly  likely 

In  items  19-23, use  the  5-point  scale  to  indicate  the  amount  that  each  of  the  following  factors  is  likely  to 
play  in  preventing  the  successful  application  of  the  IPPD  process  in  your  project: 

1  2  3  4  5 

not  at  all  a  slight  a  moderate  a  large  a  very  large 

likely  amount  amount  amount  amount 

19.  _  Too  many  unknown  or  changing  variables/conditions  prevent  estimation  of  reliable  parameters 

20.  _  Our  6.3  project  has  special  qualities  or  needs  that  are  not  amenable  to  the  normal  IPPD  process. 

21.  _  We  are  too  far  along  in  our  project  to  employ  the  process  with  any  worthwhile  effect 

22.  _  Employing  the  IPPD  process  will  take  away  time  and  resources  sorely  needed  to  complete  the 

project 
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23.  _  Add  any  other  factor  that  would  prevent  successful  application  of  the  IPPD  process 

24.  How  much  benefit  do  you  think  the  IPPD  process  could  be  to  6.3  programs  other  than  your  current 
program? 

1  2  3  4  5 

No  small  moderate  large  very  large 

benefit  benefit  benefit  benefit  benefit 

25.  To  what  extent  would  you  feel  capable  of  applying  the  IPPD  process  in  an  appropriate  program? 

1  2  3  4  5 

Not  at  Slightly  Moderately  Capable  Very 

all  capable  capable  capable 

26.  How  much  additional  training  or  mentoring  would  you  need  before  you  would  feel  capable  of 
applying  the  IPPD  process? 

1  2  3  4  5 

No  additional  A  slight  A  moderate  A  large  A  very  large 

training  amount  amount  amount  amount 

27.  How  much  does  your  management  (collectively,  not  just  any  specific  individual)  know  about  IPPD 
concepts  and  methods? 

0  1  2  3  4  5 

I  don’t  no  small  amount  moderate  amount  large  amount  very  large  amount 

know  knowledge  of  knowledge 

28.  To  what  extent  does  your  management  (collectively,  not  just  any  specific  individual)  understand  the 
reasons  for  the  IPPD  initiative? 

0  1  2  3  4  5 

I  don’t  no  slight  moderate  good  excellent 

know  understanding  understanding  understanding  understanding  understanding 

29.  To  what  extent  does  your  management  understand  the  intended  benefits  of  the  IPPD  initiative? 

0  1  2  3  4  5 

I  don’t  no  slight  moderate  good  excellent 

know  understanding  understanding  understanding  understanding  understanding 

30.  To  what  extent  does  your  management  understand  how  IPPD  methods  apply  to  your  6.3  program? 

0  1  2  3  4  5 

I  don’t  no  slight  moderate  good  excellent 

know  understanding  understanding  understanding  understanding  understanding 

3 1 .  How  likely  does  your  management  think  it  is  that  the  IPPD  initiative  can  be  successfully  applied  to 
your  6.3  program? 

0  1  2  3  4  5 

I  don’t  application  application  chances  application  application 

know  highly  unlikely  unlikely  about  50%  likely  highly  likely 
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32.  How  likely  does  your  management  think  it  is  that  IPPD  methods  can  be  applied  to  your  6.3  project 
with  real  benefits  to  the  program? 


0 

1 

2 

3 

4 

5 

I  don’t 

real  benefits 

real  benefits 

chances 

real  benefits 

real  benefits 

know 

highly  unlikely 

unlikely 

about  50  % 

likely 

highly  likely 

33.  To  what  extent  did  vour  manat 

rement  support  vour  attendance  of  this  course? 

0 

1 

2 

3 

4 

5 

I  don’t 

no 

slight 

moderate 

strong 

very  strong 

know 

support 

support 

support 

support 

support 

34.  Apart  from  any  potential  benefit  to  your  6.3  pilot  program,  to  what  extent  is  the  information  provided 
in  the  course  likely  to  be  useful  for  your  general  professional  development/advancement  in  the  future? 

1  2  3  4  5 

no  use  very  little  moderate  quite  useful  extremely 

use  use  useful 

35.  Are  all  the  government  (military  or  civil  service)  members  of  your  6.3  project  team  taking  this  course 
with  you?  If  you  answer  no,  write  in  the  approximate  percentage  of  those  who  are  attending. 

Yes _  No _  Percent  attending _ 

36.  Are  all  the  contractor  members  of  your  6.3  project  team  taking  this  course  with  you?  If  you  answer  no, 
write  in  the  approximate  percentage  of  those  who  are  attending. 

Yes _  No _  Percent  attending _ 

37.  Do  you  think  it  is  important  that  all  the  government  members  on  your  6.3  project  team  take  this  course 
at  the  same  time,  or  at  some  point? 

Yes,  at  the  same  time _  Yes,  at  some  point _  No,  not  important _ 

38.  Do  you  think  it  is  important  that  all  the  contractor  members  on  your  6.3  project  team  take  this  course 
at  the  same  time,  or  at  some  point? 

Yes,  at  the  same  time  _ _  Yes,  at  some  point _  No,  not  important  _ _ 

39.  Would  you  recommend  that  your  colleagues  (those  who  do  the  same  general  kind  of  work  you  do  but 
are  not  involved  in  a  pilot  6.3  program)  take  this  course? 

1  2  3  4  5 

would  suggest  would  suggest  it  would  strongly 

it  will  provide  is  appropriate  recommend 

little  benefit  for  some  they  take  it 

40.  Describe  briefly  any  information  you  received  about  this  course  prior  to  attendance.  Also  indicate  the 
source  of  the  information  (e.g.,  word  of  mouth,  memos,  brochures).  If  you  received  no  information,  just 
say  “none”. 
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41.  Were  you  required  to  take  this  course,  was  it  suggested  but  not  required,  or  did  you  ask  to  attend? 


Required _  Suggested,  not  required _  Asked  to 


42.  If  you  checked  either  “Required”  or  “Suggested”  for  item  4 1 ,  briefly  describe  the  reasons  you  were 
given  as  to  why  your  attendance  was  required  or  suggested. 


43.  If  you  checked  “Asked  to”  for  item  41,  briefly  say  why  you  wanted  to  attend. 


44.  Please  list  the  course  content  areas  that  you  found  to  be  the  most  useful  or  relevant  to  your  6.3  project 
(if  none  were,  say  so).  Please  be  as  specific  as  possible. 


45.  If  you  listed  any  content  areas  under  item  44,  explain  for  each  area  listed  why  you  think  it  will  be 
useful  or  relevant.  Please  be  as  specific  as  possible. 


46.  Please  list  the  course  content  areas  that  you  found  to  be  the  least  useful  or  relevant  to  your  6.3  project 
(if  all  areas  were,  say  so).  Please  be  as  specific  as  possible. 


4?-  If  you  listed  any  content  areas  under  item  46,  explain,  for  each  area  listed,  why  you  think  it  will  not 
be  useful  or  relevant.  Please  be  as  specific  as  possible. 


48.  If  you  indicated  that  any  content  areas  were  not  useful  or  relevant,  please  give  any  suggestions  you 
have  for  making  that  particular  content  more  useful  or  relevant  to  your  6.3  project.  If  any  given  area 
cannot  be  made  useful  or  relevant,  say  so. 


49.  Apart  from  anything  listed  under  item  46,  would  you  change  any  other  aspect  of  the  course  conduct  or 
content  if  you  could? 

Yes _  No _ 

50.  If  you  checked  “Yes”  for  item  49,  please  explain  as  specifically  as  possible  what  you  would  change 
and  how  you  would  change  it. 


Thank  you  very  much  for  your  assistance  in  evaluating  this  IPPD  initiative.  Please  call 
Jerry  Barucky  or  Gary  Grimes  (at  1-  800  374-6601)  with  any  questions  or  comments. 
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APPENDIX  B:  INTERVIEW  SESSION  DISCUSSION  QUESTIONS 


IPPD  FOLLOW-UP  INTERVIEW 


1.  Describe  the  manner  in  which  your  implementation  of  IPPD  has  been  organized  or  directed. 

a)  Has  your  IPT  split  into  sub-groups  to  handle  specific  issues  or  does  everyone  work  together? 

b)  Have  you  had  meetings  to  work  on  IPPD  methods,  plan  implementation,  or  discuss  status? 
How  frequently? 

c)  Are  team  members  left  on  their  own  to  implement  as  they  see  opportunities? 

d)  Are  certain  team  members  more  involved  with  IPPD  implementation  and  others  far  less 
involved?  Which  members  are  the  key  IPPD  implementers? 


2.  Are  you  satisfied  with  the  level  of  communication  within  your  IPT? 

a)  How  do  you  communicate  about  IPPD  implementation? 

b)  How  often? 

c)  What  could  be  done  to  improve? 

3.  Are  there  specific  requirements  placed  on  your  IPT  by  your  “customer”  or  “customers”  that  make 
IPPD  more  difficult  to  apply  to  your  project?  If  so,  what  are  they? 


4.  Describe  the  extent  or  the  manner  in  which  the  “users”  or  “customers”  of  the  resultant 
technology  have  been  involved  in  any  IPPD  discussions  (e.g.,  helping  lay  out  requirements 
parameters  for  the  House  of  Quality). 


5.  What  role  have  the  contractors  played  in  the  implementation  of  IPPD?  Attended  training; 
involved  in  development  of  HOQ,  technology  worksheets,  value  scorecard,  or  other  IPPD 
procedures?  Is  IPPD  implementation  a  part  of  the  deliverables  in  the  contract? 


6.  How  has  the  implementation  of  IPPD  affected  your  time  and  resources  allocated  to  the  completion 
of  the  project?  Has  it  resulted  in  savings  of  any  kind?  Has  it  taken  away  from  time  and  funds  that 
would  normally  be  devoted  to  the  task  completion? 


7.  What  beneficial  aspects  have  occurred  from  implementation  of  IPPD?  (e.g.,  more  interaction 
among  team  members?  forced  to  really  look  hard  at  performance  parameters  and  consider  a  broader 
set  of  perspectives?  more  careful  consideration  of  potential  alternatives?  knowledge  gained  in 
designing  experiments  for  testing?) 
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8.  Is  it  clear  what  specific  actions  and  outcomes  you  should  be  able  to  demonstrate  as  a  result  of  the 
implementation  of  IPPD? 

a)  Do  you  think  the  Air  Force  should  specifically  state  what  they  expect  from  your  IPT  regarding 
IPPD;  or  would  you  rather  set  expectations  yourself? 

b)  Do  you  understand  the  Air  Force’s  expectations? 

c)  What  do  you  think  the  Air  Force  expects  out  of  the  IPPD  movement? 


9.  What  criteria  does  your  team  currently  use  to  measure  performance  and/or  success  of  IPPD 
implementation? 


10.  Tell  me  what  IPPD  procedures  and  tools  you’ve  actually  used  in  your  6.3  pilot  project. 

a)  Have  you  completed  an  HOQ?  Are  you  working  on  one? 

b)  Has  your  project  identified  alternative  methods  or  processes  or  parameters  that 

might  result  in  tradeoffs  in  performance,  cost,  producibility,  maintainability,  etc.  Have  you  collected 
data  or  run  experiments?  Have  you  developed  performance,  cost,  or  producibility  worksheets  to 
document  examination  of  those  alternatives? 

c)  Has  your  IPT  developed  a  Value  Scorecard  to  summarize  the  results  of  the  consideration  of 
alternatives? 

d)  Describe  a  specific  example  in  which  the  use  of  IPPD  processes  was  particularly  helpful  or  in 
which  they  were  applied  successfully. 

e)  Describe  a  specific  example  in  which  the  attempted  application  of  IPPD  processes  was 
particularly  difficult  or  was  not  successful . 


1 1.  To  what  extent  is  the  technical  support  needed  for  implementation  of  IPPD  procedures?  (Does 
team  make  progress  in  IPPD  without  support  of  Peisert  or  other  contracted  advisors,  or  is  their  help 
essential  at  this  point  in  the  building  of  HOQ,  development  of  worksheets,  etc.?) 


12.  When  the  questionnaire  asked  about  management  support,  what  level  of  management  were  you 
referring  to:  Division,  Directorate,  Lab,  or  some  other? 


13.  What  are  the  probable  outcomes  of  this  overall  IPPD  initiative  in  the  S&T  community?  (Do  you 
think  IPPD  techniques  will  lead  to  actual  increases  in  efficiencies  and  effectiveness  in  your  or  other 
projects?  is  it  just  a  tool  to  measure  and  document  what  you  are  currently  doing  without  changing  the 
outcomes?,  or  is  it  likely  to  be  a  temporary  managerial  issue  that  will  eventually  fade  in  importance?) 
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APPENDIX  C:  FOLLOW-UP  QUESTIONNAIRE 


Follow-up  Questionnaire 


Name:  _ 

Organization:  _ 

Commercial  Telephone: 
Email  Address:  _ 

Date  Course  was  taken: 


Instructions:  This  questionnaire  gathers  information  that  will  be  used  to  evaluate  and 
refine  the  Affordable  Technology/Integrated  Product  &  Process  Development,  AT/IPPD, 
(previously  known  as  Design  for  Six  Sigma  Manufacturing,  or  DSSM)  initiative  in  the  Air 
Force’s  Science  and  Technology  (S&T)  environment.  For  the  purposes  of  this  evaluation, 
IPPD  will  refer  to  both  DSSM  and  AT/IPPD.  A  full  evaluation  effort  requires  extensive 
information  not  only  about  the  use  and  effectiveness  of  training  and  mentoring  programs, 
but  also  about  how  and  how  well  the  IPPD  process  is/has  been  implemented  across  pilot 
projects  and  the  obstacles  that  make  the  process  difficult  to  implement.  We,  therefore, 
requested  verification  of  your  name  and  other  information  so  that  we  can  follow  up  on  this 
questionnaire  if  necessary.  We  will  not  provide  or  report  identifiable  individual  responses 
to  anyone.  Select  the  MOST  correct  response.  Whenever  an  item  refers  to  “your  6.3 
project,”  assume  that  it  refers  to  the  pilot  6.3  project  to  which  the  IPPD  process  is  being 
applied.  If  you  are  not  a  pilot  team  member,  assume  it  refers  to  your  current  project. 
Place  additional  write-in  comments  at  the  end  of  this  questionnaire. 


1 .  Are  you  currently  either  working  on  or  in  some  substantial  way  connected  with  one  of  the 
IPPD  6.3  pilot  projects? 

Yes  No 


la.  Answer  only  if  you  answered  “no”  to  question  1:  Since  you  are  not  currently  working  on  or 
in  some  substantial  way  connected  with  one  of  the  IPPD  6.3  pilot  projects,  please  briefly 
explain  why  you’re  no  longer  with  the  project,  or  why  you  previously  attended  an  IPPD 
training  course. 
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lb.  Which  pilot  program  are  you  currently  assigned  to  as  an  integrated  product  team  member? 
(circle) 


a. 

3  DOC 

h. 

ISCP 

b 

AMD 

i. 

MASS 

c. 

BWS 

j- 

ALEP 

d. 

RCM 

k. 

LCCAAS 

e. 

STD 

1. 

VEIITS 

f. 

NGT 

m. 

OHRDTP 

g- 

CAI 

n. 

Other 

o. 

Not  a  member  of  any  IPPD  pilot  program 

2.  Which  response  best  describes  your  employment  category? 

Government _  Contractor _  Other _ Please  explain  briefly  below 

(Military  or  civil  service)  (If  you  consider  yourself  a  customer  or  user  for 

6.3  program,  please  say  so.) 


3.  Which  of  the  following  best  describes  your  current  position  relative  to  the  IPPD  project 
you’re  working  on  (circle  one): 


a.  Technical  Manager 

b.  Technical  Supervisor 

c.  Administrative  Manager 

d.  Administrative  Supervisor 

e.  Researcher 

f.  Bench  Level  Engineer 

g.  Scientist 

d.  Contracting  Officer 

e.  Other  (please  describe  briefly): 


4.  Indicate  the  extent  of  your  present  knowledge  about  IPPD  concepts  and  methods. 

1  2  3  4  5 

No  Small  amount  Moderate  amount  Large  amount  Very  large  amount 

knowledge  of  knowledge 

5.  To  what  extent  do  you  presently  understand  the  reasons  for  the  IPPD  initiative? 

1  2  3  4  5 

No  Slight  Moderate  Good  Excellent 

understanding  understanding  understanding  understanding  understanding 
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6.  To  what  extent  do  you  presently  understand  the  intended  benefits  of  the  IPPD  initiative  ? 


1  2  3  4  5 

No  Slight  Moderate  Good  Excellent 

understanding  understanding  understanding  understanding  understanding 


7.  To  what  extent  do  you  presently  understand  how  IPPD  methods  apply  to  your  6.3  program? 

1  2  3  4  5 

No  Slight  Moderate  Good  Excellent 

understanding  understanding  understanding  understanding  understanding 


8.  In  general,  to  what  extent  has  vour  team  been  able  to  apply  the  IPPD  process,  methods,  and 
techniques  to  your  project? 

0  1  2  3  4  5 

Don’t  know  Not  at  all  Slight  Moderate  Large  Very  large  extent 


9.  To  what  extent  have  von  participated  in  the  overall  application  of  the  IPPD  process  to  your 
project? 

1  2  3  4  5 

Not  at  all  Slight  Moderate  Large  Very  large  extent 


10.  To  what  extent  has  vour  team  completed  a  house  of  quality,  HOQ,  for  your  IPPD  project? 

0  1  2  3  4  5 

Don’t  know  Have  not/  Quarter,  25%  Half,  50%  Three  Quarters,  75%  Completed 
Just  begun 


1 1.  To  what  extent  did  you  participate  in  completing  a  house  of  quality,  HOQ,  for  your  IPPD 


project? 

1 

2 

3 

4 

5 

Did  not 

A  slight 

Moderately 

Significantly 

Most  prominent 

amount 

contributor 
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12.  To  what  extent  has  your  team  used  technology  worksheets  to  develop/investigate  potential 
alternatives  in  terms  of  their  costs  and  likelihood  of  success,  producibility  or  failure? 


0  1  2  3  4  5 

Don’t  know  None  of  A  little  Half  of  Most  of  Always 

the  time  of  the  time  the  time  the  time 


13.  To  what  extent  did  you  participate  in  using  technology  worksheets  to  develop/investigate 
potential  alternatives  in  terms  of  their  costs  and  likelihood  of  success,  producibility,  or  failure? 

1  2  3  4  5' 

Did  not  A  slight  amount  Moderately  Significantly  Most  prominent  contributor 


14.  To  what  extent  has  your  team  used  value  scorecards  to  determine  the  best  alternatives  in 
terms  of  their  likelihood  of  success,  producibility,  or  failure? 


0 

1 

2 

3 

4 

5 

Don’t 

Know 

None  of 
the  time 

A  little 
of  the  time 

Half  of 
the  time 

Most  of 
the  time 

Always 

15.  To  what  extent  have  you  participated  in  using  value  scorecards  to  determine  the  best 
alternatives  in  terms  of  their  likelihood  of  success,  producibility,  or  failure? 

1  2  3  4  5 

Did  not  A  slight  amount  Moderately  Significantly  Most  prominent  contributor 


16.  To  what  extent  has  the  use  of  IPPD  methods  helped  vour  team  reduce  costs? 

0  1  2  3  4  5 

Don’t  know  Not  at  all  Slightly  Moderately  Large  extent  Very  large  extent 


17.  To  what  extent  has  the  use  of  IPPD  methods  helped  you  reduce  costs? 

1  2  3  4  5 

Not  at  all  Slightly  Moderately  Large  Very  large  extent 
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18.  To  what  extent  has  the  use  of  IPPD  methods  helped  your  team  shorten  timelines? 


0  1  2  3  4  5 

Don’t  know  Not  at  all  Slightly  Moderately  Large  Very  large  extent 


19.  To  what  extent  has  the  use  of  IPPD  methods  helped  you  shorten  timelines? 

1  2  3  4  5 

Not  at  all  Slightly  Moderately  Large  Very  large  extent 


20.  To  what  extent  has  the  use  of  IPPD  methods  helped  your  team  increase  productivity? 

0  1  2  3  4  5 

Don’t  know  Not  at  all  Slightly  Moderately  Large  Very  large  extent 


21.  To  what  extent  has  the  use  of  IPPD  methods  helped  vow  increase  productivity? 

1  2  3  4  5 

Not  at  all  Slightly  Moderately  Large  Very  large  extent 

22.  To  what  extent  has  the  use  of  IPPD  methods  helped  vour  team  identity  risk  associated  with 
the  project? 

0  1  2  3  4  5 

Don’t  know  Not  at  all  Slightly  Moderately  Large  Very  large  extent 

23.  To  what  extent  has  the  use  of  IPPD  methods  helped  you  identify  risk  associated  with  the 
project? 

1  2  3  4  5 

Not  at  all  Slightly  Moderately  Large  Very  large  extent 


24.  To  what  extent  are  you  aware  of  IPPD  efforts/progress  made  by  your  IPT  team  members? 

1  2  3  4  5 

Not  at  all  Slightly  Moderately  Large  Very  large  extent 
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In  items  25-32,  use  the  5-point  scale  below  to  indicate  the  amount  that  each  of  the  following 
factors  played  in  hindering  successful  application  of  the  IPPD  process  in  your  team  project: 

12  3  45 

Not  at  all  A  slight  amount  A  moderate  amount  A  large  amount  A  very  large  amount 

25.  _ Too  many  unknown  or  changing  variables/conditions 

26.  _ Our  6.3  project  has  special  qualities  or  needs  that  are  not  amenable  to  the  normal  IPPD 

process. 

27.  _ We  were  too  far  along  in  our  project  to  employ  the  process  with  any  worthwhile  effect. 

28.  _  Not  enough  time. 

29.  _  Not  have  enough  knowledge  or  training  on  IPPD  methodologies. 

30.  _  Not  enough  resources/materials  available. 

31.  _  Not  enough  support/emphasis  from  management. 

32.  _  Write  in,  then  rate,  any  other  factor(s)  that  prevented  successful  application  of  the 

IPPD  process. 


33.  At  what  phase  of  a  project  do  you  believe  it  would  be  most  beneficial  to  implement  IPPD 
methods? 


1 

2 

3 

4 

5 

Start  up 
0-20% 

21-40% 

Halfway 

41-60% 

61-80% 

Project  Wrap-up 

81-1 00%  Completion 

34.  How  much  benefit  do  you  think  the  IPPD  process  could  be  to  6.3  programs  other  than 
your  current  program? 

1  2  3  4  5 

No  benefit  Small  benefit  Moderate  benefit  Large  benefit  Very  large  benefit 


35.  How  much  does  your  management  (collectively,  not  just  any  specific  individual)  know 
about  IPPD  concepts  and  methods? 

0  1  2  3  4  5 

I  don’t  No  Small  amount  Moderate  amount  Large  amount  Very  large  amount 
know  knowledge  of  knowledge 
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36.  To  what  extent  does  your  management  (collectively,  not  just  any  specific  individual) 
understand  the  reasons  for  the  IPPD  initiative? 

0  1  2  3  4  5 

I  don’t  No  Slight  Moderate  Good  Excellent 

know  understanding  understanding  understanding  understanding  understanding 


37.  To  what  extent  does  your  management  understand  the  intended  benefits  of  the  IPPD 


—  -  «/ 

initiative? 

0  1 

o 

2 

3 

4 

5 

I  don’t 

No 

Slight 

Moderate 

Good 

Excellent 

know 

understanding 

understanding 

Understanding 

understanding 

understanding 

38.  To  what  extent  does  your  management  understand  how  IPPD  methods  apply  to  your  6.3 
program? 

0  1  2  3  4  5 

I  don’t  No  Slight  Moderate  Good  Excellent 

know  understanding  understanding  Understanding  understanding  understanding 


39.  At  the  time  you  received  your  IPPD  training,  how  did  your  management  perceive  that  the 
IPPD  initiative  could  be  successfully  applied  to  your  6.3  program? 


0  1  2  3 

I  don’t  Application  Application  Chances 
know  highly  unlikely  unlikely  about  50% 


4 

Application 

likely 


5 

Application 
highly  likely 


40.  How  does  your  management  now  think  the  IPPD  initiative  has  been  applied  to  your  6.3 
program? 


0  1 
I  don’t  Application 
know  nonexistent 


2 

Application 

weak 


3 

Application 

mediocre 


4  5 

Application  Application 
successful  highly  successful 


41.  At  the  time  you  received  your  IPPD  training,  what  were  your  management’s  perceptions 
about  IPPD  methods  resulting  in  real  benefits  to  the  6.3  project? 


0  1  2 
I  don’t  Real  benefits  Real  benefits 

know  highly  unlikely  unlikely 


3  4 

Chances  Real  benefits 
about  50  %  likely 


5 

Real  benefits 
highly  likely 


61 


42.  To  what  extent  does  your  management  believe  that  the  IPPD  process  has  brought  real 
benefits  to  the  project? 

0  1  2  3  4  5 

I  don’t  No  Small  Moderate  Large  Very  large 

know  benefit  benefit  benefit  benefit  benefit 


43.  To  what  extent  did  management  support  your  attendance  at  your  initial  IPPD  training? 

0  1  2  3  4  5 

I  don’t  No  Slight  Moderate  Strong  Very  strong 

know  support  support  support  support  support 


44.  How  much  has  your  management  actually  supported  your  efforts  to  use  IPPD  methods? 

0  1  2  3  4  5 

I  don’t  No  Slight  Moderate  Strong  Very  strong 

know  support  support  support  support  support 

45.  How  much  has  your  management  encouraged  you  to  use  mentoring  and/or  take  the 
available  supplemental  EPPD  courses? 

0  1  2  3  4  5 

I  don’t  No  Slight  Moderate  Strong  Very  strong 

know  support  support  support  support  support 

46.  How  likely  is  it  that  you  will  use  IPPD  methods  on  future  projects? 

12  34  5 

Application  Application  Chances  Application  Application 

highly  unlikely  unlikely  about  50%  likely  highly  likely 

47.  To  what  extent  do  you  feel  capable  of  applying  the  IPPD  process  in  an  appropriate  program? 

1  2  3  4  5 

Not  at  Slightly  Moderately  Capable  Very 

all  capable  capable  capable 

48.  How  much  additional  training  or  mentoring  would  you  need  before  you  would  feel  capable 
of  applying  the  IPPD  process. 

1  2  3  4  5 

No  additional  A  slight  A  moderate  A  large  A  very  large 
training  amount  amount  amount  amount 
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49.  What  other  IPPD  courses  have  you  taken?  (circle  all  that  apply) 


a.  None 

b.  Design  To  Cost  (DTC) 

c.  Design  Of  Experiment  (DOE) 

d.  Design  for  Six  Sigma  Manufacturability  (DSSM) 

e.  Other 


50.  Please  estimate  the  total  number  of  times  and  hours  you’ve  received  mentorship  from  the 
course  instructors. 

Number  of  mentorship  communications _  Total  hours  of  assistance _ 

51.  Apart  from  any  potential  benefit  to  your  6.3  pilot  program,  to  what  extent  was  the 
information  provided  in  the  course  useful  for  your  general  professional  development/ 
advancement  in  the  future? 

1  2  3  4  5 

No  use  Very  little  use  Moderate  use  Quite  useful  Extremely  useful 

52.  Have  all  the  government  (military  or  civil  service)  members  of  your  6.3  project  team  taken 
the  course?  If  you  answer  no,  write  in  the  approximate  percentage  of  those  who  have  attended. 

Yes _  No _  Percent  attended _ 

53.  Have  all  the  contractor  members  of  your  6.3  project  team  taken  the  course?  If  you 
answer  no,  write  in  the  approximate  percentage  of  those  who  have  attended. 

Yes _  No _  Percent  attended _ 

54.  Do  you  think  it  is  important  that  all  the  government  members  on  your  6.3  project  team  take 
the  IPPD  course  at  the  same  time,  or  at  some  point? 

Yes,  at  the  same  time _  Yes,  at  some  point _  No,  not  important _ 

55.  Do  you  think  it  is  important  that  all  the  contractor  members  on  your  6.3  project  team 
take  this  course  at  the  same  time  as  the  government  members,  or  at  some  point? 

Yes,  at  the  same  time _  Yes,  at  some  point _  No,  not  important _ 

56.  Do  you  think  it  is  important  that  all  team  members  attend  IPPD  courses  or  just  certain  “key” 
members? 

AH  team  members _  Key  individuals  only _ 
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57.  Would  you  recommend  that  your  colleagues  (those  who  do  the  same  general  kind  of  work 
you  do  but  are  not  involved  in  a  pilot  6.3  program)  take  this  course? 


0  1  2 

No  Yes, 

it  will  provide 
a  little  benefit 


3  4  5 

Yes,  it  Yes,  strongly 

is  appropriate  recommend 

for  some  they  take  it 


58.  Who  is/would  be  in  the  best  position  to  evaluate  IPPD  effectiveness  in  your  project? 

Team  members _  Program  Manager _  Branch/Division  Chief _ 

Lab  Director _  Other 


59.  Do  you  feel  your  program  has  benefited  from  using  IPPD  methods? 

1  2  3  4  5 

No  Slightly  Moderately  Greatly  Excellently 

Briefly  explain  why/how  you  think  your  program  has  benefited. 

60.  How  did  you  measure  the  product/process  improvements  attributed  to  IPPD  methods  used  in 
your  project  as  compared  to  other  methods  used? 


61 .  Has  the  working  environment  or  processes  where  you  work  changed  as  a  result  of  your  IPPD 
training? 

_ No  _ Yes  (If  yes,  briefly  state  how) 


Thank  you  very  much  for  your  assistance  in  evaluating  this  IPPD  initiative.  Your  input  will 
prove  invaluable  to  our  evaluation.  Please  call  Jerry  Barucky  or  Gary  Grimes  at  1-800  374-6601 
with  any  questions  or  additional  comments. 

COMMENTS 
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APPENDIX  D:  SUMMARY  OF  PROCESSES  USED  SURVEY 


SUMMARY  OF  PROCESSES  USED  IN  PILOT  IPT 

(In  each  area,  circle  the  option  that  best  describes  the  process  experiences  or  intentions  for  your  pilot  project.  If 
none  of  the  options  sufficiently  describe  your  situation,  please  write  a  short  alternative  description.) 


User/Customer  Interaction  in  IPT  Activities 

1.  Requirements  determined  from  previous  interaction  in  research  area;  little  additional  customer  involvement 

2.  Requirements  outlined/discussed  at  general  meetings  and  through  documents  outlining  general  needs  (e.g., 
TPIPTs  or  deficiency  documents  such  as  ORDS);  little  continuous  involvement  in  actual  HOQ  development  or 
other  requirements  documentation  sessions 

3.  Involved  in  determining  requirements  through  general  means  described  above  and  at  initial  specific  meetings 
regarding  the  technology  project;  little  continuous  involvement  in  development  of  HOQ,  Exit  Criteria,  or  other 
requirements  documentation  sessions 

4.  Continuous  involvement  at  initial  sessions  and  in  development  of  HOQ  or  other  requirements  documents 

a.  little  continuous  involvement  after  this 

b.  continuous  involvement  planned,  but  project  not  there  yet 

5.  Continuous  involvement  in  most  aspects  of  IPT,  even  after  requirements  and  or  exit  criteria  were  established 


Clear  Determination  of  Requirements 

1.  General  requirements  outlined  from  needs  documents  or  meetings,  but  no  formal  documentation 

2.  Formal  performance  requirements  developed  and  documented,  but  no  exit  criteria  documented 

3.  Formal  performance  requirements  developed  and  documented,  and  technology/engineering  requirements  also 
outlined  and  compared,  but  no  exit  criteria  formally  documented 

4.  Formal  performance  requirements  developed  and  documented,  technology/engineering  requirements  outlined, 
and  exit  criteria  formally  documented 

House  of  Quality  (HOQ)  Documented 

1.  No  House  of  Quality  developed  in  determining  requirements 

2.  Process  for  determining  requirements  used  that  had  similar  result  as  HOQ  (please  describe  or  give  example) 

3.  HOQ  begun  and  still  working  on  it 

4.  HOQ  completed 
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Technology/Design  Alternatives/Options  Compared 

1.  Decisions  regarding  technology/design  alternatives  based  on  assessments  of  experienced  researchers,  or  current 
knowledge;  little  testing  or  data  gathering  required 

2.  Decisions  regarding  technology/design  alternatives  based  on  tests,  experiments,  and  data  gathered;  little  formal 
documentation  of  results 

3.  Decisions  regarding  technology/design  alternatives  based  on  tests,  experiments,  and  data  gathered;  formal 
documentation  using  methods  other  than  IPPD  Technology  Worksheets  (please  describe  method) 


4.  Decisions  regarding  technology/design  alternatives  based  on  tests,  experiments,  and  data  gathered;  formal 
documentation  using  IPPD  Technology  Worksheets 

5.  Project  not  at  this  point  yet,  but  intentions  are  most  likely  to  reflect  description  1  2  3  4  above  (please  circle) 
Value  Analysis 


1.  Decisions  based  on  general  intuitive  considerations  of  information  relating  to  alternatives’  pros  and  cons;  no 
formal  documentation  of  analysis  process 

2.  Decisions  based  on  general  intuitive  considerations  of  information  relating  to  alternatives’  pros  and  cons; 
documentation  of  analysis  process  used  methods  other  than  Value  scorecards  (please  describe  methods) 


3.  Decisions  based  on  data  relating  to  alternatives’  performance,  producibility,  costs,  etc.;  documentation  of 
analysis  process  used  methods  other  than  Value  Scorecards  (please  describe  methods) 


4.  Decisions  based  on  data  relating  to  alternatives’  performance,  producibility,  costs,  etc.;  documentation  of 
analysis  process  included  use  of  Value  Scorecards 

5.  Project  not  at  this  point  yet,  but  intentions  are  most  likely  to  reflect  description  1  2  3  4  above  (please  circle) 

Technology  Transition  Business  Plan  (TTBP) 

1.  TTBP  not  developed 

2.  TTBP  developed  summarizing  cost,  risk,  benefits,  and  transition  assessments  arrived  at  through  Value  Scorecard, 
HOQ  requirements,  and  other  IPPD  products 

3.  TTBP  developed  summarizing  cost,  risk,  benefits,  and  transition  assessments  arrived  at  by  means  other  than 
IPPD  products 

4.  Project  not  at  this  point  yet,  but  intentions  are  most  likely  to  reflect  description  1  2  3  above  (please  circle) 
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IPT 

AMD 

ISCP 

WINDS 

IRCM 

OTH 

NGT 

MASS 

ALEP 

CAI 

VIVIDS 


IPT  IPPD  IMPLEMENTATION  SUMMARY 
IPPD  PROCESS/PROCEDURES 

MGT  CUST  RQMTS/  HOQ  COMP  Alts/ 3  VAL  Anal  Devel/  Devel 

SUPRT  INTRA  EXITCRI  Tech  Wksheet  Scorecd  Demo  TTBP 
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APPENDIX  E 

Tables  of  Questionnaire  Responses  by  Pilot  Group 
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Table  E-l.  Percent  by  Pilot  Reporting  Various  Amounts  of  IPPD  Experience 


Values 

1 

2 

3 

4 

Pilots 

5  6 

7 

8 

9 

10 

None  to  Small 

67% 

72% 

14% 

60% 

0% 

86% 

50% 

71% 

29% 

67% 

Moderate 

22% 

20% 

34% 

40% 

100% 

0% 

33% 

29% 

29% 

33% 

Large  to  Very  Large 

11% 

8% 

52% 

0% 

0% 

14% 

17% 

0% 

42% 

0% 

Table  E-2.  Percent  by  Pilot  Reporting  Various  Amounts  of  Pre-Training  IPPD  Knowledge 


Values 

1 

2 

3 

4 

Pilots 

5  6 

7 

8 

9 

10 

None  to  Small 

78% 

68% 

28% 

80% 

100% 

43% 

29% 

100% 

71% 

100% 

Moderate 

11% 

24% 

48% 

20% 

0% 

28% 

57% 

0% 

15% 

0% 

Large  to  Very  Large 

11% 

8% 

24% 

0% 

0% 

29% 

14% 

0% 

14% 

0% 

Table  E-3.  Percent  by  Pilot  Reporting  Various  Amounts  of  Post-Training  IPPD  Knowledge 


Values 

1 

2 

3 

4 

Pilots 

5  6 

7 

8 

9 

10 

None  to  Small 

0% 

16% 

3% 

0% 

0% 

29% 

0% 

14% 

0% 

0% 

Moderate 

67% 

52% 

45% 

80% 

100% 

14% 

29% 

57% 

71% 

67% 

Large  to  Very  Large 

33% 

32% 

52% 

20% 

0% 

57% 

71% 

29% 

29% 

33% 

Table  E-4.  Percent  by  Pilot  Reporting  Various  Amounts  of  Pre-Training  Understanding  of  IPPD  Initiative 


Values 

1 

2 

3 

4 

Pilots 

5  6 

7 

8 

9 

10 

None  to  Slight 

33% 

64% 

38% 

40% 

100% 

29% 

43% 

100% 

43% 

33% 

Moderate 

33% 

24% 

28% 

40% 

0% 

42% 

14% 

0% 

14% 

33% 

Good  to  Excellent 

34% 

12% 

34% 

20% 

0% 

29% 

43% 

0% 

43% 

34% 
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Table  E-5.  Percent  by  Pilot  Reporting  Various  Amounts  of  Post-Training  Understanding  of  IPPD 
Initiative  


Values 

1 

2 

3 

4 

Pilots 

5  6 

7 

8 

9 

10 

None  to  Slight 

0% 

12% 

3% 

0% 

0% 

0% 

0% 

0% 

0% 

0% 

Moderate 

33% 

36% 

31% 

20% 

100% 

43% 

0% 

43% 

43% 

33% 

Good  to  Excellent 

67% 

52% 

66% 

80% 

0% 

57% 

100% 

57% 

57% 

67% 

Table  E-6.  Percent  by  Pilot  Reporting  Various  Amounts  of  Pre-Training  Understanding  of  IPPD  Benefits 


Values 

1 

2 

3 

4 

Pilots 

5  6 

7 

8 

9 

10 

None  to  Slight 

33% 

72% 

48% 

60% 

100% 

29% 

43% 

100% 

57% 

33% 

Moderate 

44% 

16% 

21% 

20% 

0% 

28% 

14% 

0% 

0% 

33% 

Good  to  Excellent 

23% 

12% 

31% 

20% 

0% 

43% 

43% 

0% 

43% 

34% 

Table  E-7.  Percent  by  Pilot  Reporting  Various  Amounts  of  Post-Training  Understanding  of  IPPD  Benefits 


Values 

1 

2 

3 

4 

Pilots 

5  6 

7 

8 

9 

10 

None  to  Slight 

0% 

8% 

7% 

0% 

0% 

14% 

0% 

0% 

0% 

0% 

Moderate 

56% 

32% 

38% 

20% 

100% 

29% 

0% 

43% 

43% 

33% 

Good  to  Excellent 

44% 

60% 

55% 

80% 

0% 

57% 

100% 

57% 

57% 

67% 

Table  E-8.  Percent  by  Pilot  Reporting  Various  Amounts  of  Pre-Training  Understanding  of  BPPD 
Application 


Values 

1 

2 

3 

4 

Pilots 

5  6 

7 

8 

9 

10 

None  to  Slight 

78% 

76% 

62% 

80% 

100% 

57% 

57% 

100% 

33% 

67% 

Moderate 

0% 

24% 

31% 

20% 

0% 

28% 

0% 

0% 

33% 

33% 

Good  to  Excellent 

22% 

0% 

7% 

0% 

0% 

14% 

43% 

0% 

34% 

0% 
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Table  E-9.  Percent  by  Pilot  Reporting  Various  Amounts  of  Post-Training  Understanding  of  IPPD 
Application _ 


Values 

1 

2 

3 

4 

Pilots 

5  6 

7 

8 

9 

10 

None  to  Slight 

22% 

12% 

21% 

20% 

0% 

14% 

0% 

57% 

17% 

0% 

Moderate 

33% 

60% 

38% 

60% 

100% 

29% 

0% 

28% 

0% 

67% 

Good  to  Excellent 

45% 

28% 

41% 

20% 

0% 

37% 

100% 

14% 

83% 

33% 

Table  E-10.  Percent  by  Pilot  Reporting  Pre-Training  Perceptions  of  Likelihood  of  Successful  IPPD 
Application 

Values  Pilots 


1 

2 

3 

4 

5 

6 

7 

8 

9 

10 

No  Opinion 

22% 

36% 

31% 

20% 

100% 

14% 

14% 

29% 

33% 

33% 

Highly  Unlikely  to 
Unlikely 

44% 

20% 

24% 

20% 

0% 

14% 

15% 

71% 

33% 

0% 

50% 

22% 

24% 

31% 

40% 

0% 

43% 

0% 

0% 

17% 

67% 

Likely  to  Highly 
Likely 

12% 

20% 

14% 

20% 

0% 

29% 

71% 

0% 

17% 

0% 

Table  E-ll.  Percent  by  Pilot  Reporting  Post-Training  Perceptions  of  Likelihood  of  Successful  IPPD 
Application _ 

Values  ~  Pilots 


1 

2 

3 

4 

5 

6 

7 

8 

9 

10 

Highly  Unlikely  to 
Unlikely 

38% 

0% 

14% 

0% 

100% 

0% 

0% 

100% 

20% 

0% 

50% 

37% 

54% 

52% 

60% 

0% 

33% 

14% 

0% 

0% 

33% 

Likely  to  Highly 
Likely 

25% 

46% 

34% 

40% 

0% 

67% 

86% 

0% 

80% 

67% 
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Table  £-12.  Percent  by  Pilot  Reporting  Pre-Training  Perceptions  of  Likelihood  of  Achieving 
Benefits  from  IPPD 


Values 

1 

2 

3 

4 

Pilots 

5  6 

7 

8 

9 

10 

No  Opinion 

11% 

44% 

14% 

20% 

0% 

14% 

14% 

29% 

20% 

33% 

Highly  Unlikely  to 
Unlikely 

55% 

20% 

31% 

40% 

100% 

14% 

29% 

71% 

20% 

0% 

50% 

22% 

24% 

38% 

40% 

0% 

43% 

0% 

0% 

20% 

67% 

Likely  to  Highly 
Likely 

12% 

12% 

17% 

20% 

0% 

29% 

57% 

0% 

40% 

0% 

Table  E-13.  Percent  by  Pilot  Reporting  Post-Training  Perceptions  of  Likelihood  of  Achieving  Benefits  from 
BPPD 


Values 

1 

2 

3 

4 

Pilots 

5  6 

7 

8 

9 

10 

Highly  Unlikely  to 
Unlikely 

25% 

8% 

10% 

0% 

100% 

0% 

0% 

100% 

20% 

0% 

50% 

50% 

67% 

52% 

40% 

0% 

43% 

14% 

0% 

0% 

33% 

Likely  to  Highly 
Likely 

25% 

25% 

38% 

60% 

0% 

57% 

86% 

0% 

80% 

67% 

Table  E-14.  Percent  by  Pilot  Reporting  Post-Training  Perceptions  of  Potential  IPPD  Benefits  to  Other  Project 


Values 

1 

2 

3 

4 

Pilots 

5 

6 

7 

8 

9 

None  to  Small 

22% 

0% 

3% 

0% 

0% 

0% 

0% 

0% 

0% 

Moderate 

56% 

74% 

45% 

0% 

100% 

29% 

43% 

50% 

0% 

Large  to  Very  Large 

22% 

26% 

52% 

100% 

0% 

71% 

57% 

50% 

100 
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Table  E-15.  Percent  by  Pilot  Reporting  Post-Training  Perceptions  of  Capability  of  Applying  IPPD 


Values 

Pilots 

1 

2 

3 

4 

5 

6 

7 

8 

9  * 

Not  to  Slight 

25% 

25% 

21% 

60% 

100% 

29% 

14% 

29% 

14% 

Moderate 

38% 

54% 

48% 

40% 

0% 

28% 

43% 

43% 

72% 

Capable  to  Very 
Capable 

37% 

21% 

31% 

0% 

0% 

43% 

43% 

29% 

14% 

Table  E-16.  Percent  by  Pilot  Reporting  Post-Training  Perceptions  of  IPPD  Additional  Training  or  Mentoring 
Needed 


Values 

Pilots 

1 

2 

3 

4 

5 

6 

7 

8 

9 

None  to  Small 

22% 

8% 

14% 

20% 

0% 

43% 

29% 

14% 

14% 

Moderate 

45% 

67% 

62% 

60% 

0% 

28% 

71% 

29% 

72% 

Large  to  Very  Large 

33% 

25% 

24% 

20% 

100% 

29% 

0% 

57% 

14% 

Table  E-17.  Percent  by  Pilot  Reporting  Post-Training  Perceptions  of  the  Usefulness  of  IPPD  For  Professional 
Development _ 


Values 

Pilots 

1 

2 

3 

4 

5 

6 

7 

8 

9 

No  Use 

0% 

0% 

0% 

0% 

0% 

0% 

0% 

0% 

0% 

Very  Little 

25% 

11% 

0% 

0% 

0% 

0% 

6% 

29% 

20% 

Moderate 

50% 

22% 

39% 

20% 

100% 

60% 

14% 

0% 

20% 

Quite  Useful  to 
Extremely  Useful 

25% 

67% 

61% 

80% 

0% 

40% 

86% 

71% 

60% 
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Table  E-18.  Percent  by  Pilot  Reporting  Post-Training  Perceptions  of  Management’s  Knowledge 
of  IPPD  Process 


Values 

1 

2 

3 

4 

Pilots 

5  6 

7 

8 

9 

10 

Don’t  Know 

0% 

17% 

7% 

0% 

0% 

0% 

14% 

0% 

14% 

0% 

None  to  Small 

33% 

50% 

65% 

60% 

100% 

29% 

71% 

57% 

57% 

100% 

Moderate 

56% 

25% 

21% 

40% 

0% 

28% 

15% 

43% 

29% 

0% 

Large  to  Very  Large 

11% 

8% 

7% 

0% 

0% 

43% 

0% 

0% 

0% 

0% 

Table  E-19.  Percent  by  Pilot  Reporting  Post-Training  Perceptions  of  Management’s  Understanding  of  IPPD 


Initiative 


Values 

I 

2 

3 

4 

Pilots 

5  6 

7 

8 

9 

10 

Don’t  Know 

0% 

21% 

7% 

0% 

0% 

0% 

0% 

0% 

28% 

0% 

None  to  Slight 

44% 

50% 

48% 

40% 

100% 

14% 

57% 

43% 

43% 

100% 

Moderate 

34% 

8% 

28% 

60% 

0% 

29% 

43% 

28% 

0% 

0% 

Good  to  Excellent 

22% 

21% 

17% 

0% 

0% 

57% 

0% 

29% 

29% 

0% 

Table  E-20.  Percent  by  Pilot  Reporting  Post-Training  Perceptions  of  Management’s  Understanding  of  IPPD 


Benefits 


Values 

1 

2 

3 

4 

Pilots 

5  6 

7 

8 

9 

10 

Don’t  Know 

0% 

21% 

11% 

0% 

0% 

0% 

0% 

0% 

28% 

0% 

None  to  Slight 

44% 

40% 

41% 

60% 

100% 

0% 

57% 

43% 

43% 

67% 

Moderate 

34% 

13% 

31% 

20% 

0% 

43% 

43% 

43% 

14% 

0% 

Good  to  Excellent 

22% 

21% 

17% 

20% 

0% 

57% 

0% 

14% 

15% 

33% 
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Table  E-21.  Percent  by  Pilot  Reporting  Post-Training  Perceptions  of  Management’s 
Understanding  of  IPPD  Application 


Values 

1 

2 

3 

4 

Pilots 

5  6 

7 

8 

9 

10 

Don’t  Know 

0% 

21% 

21% 

0% 

0% 

0% 

0% 

0% 

29% 

0% 

None  to  Slight 

50% 

50% 

69% 

60% 

100% 

0% 

86% 

71% 

57% 

100% 

Moderate 

25% 

16% 

10% 

20% 

0% 

57% 

14% 

0% 

0% 

0% 

Good  to  Excellent 

25% 

13% 

0% 

20% 

0% 

43% 

0% 

29% 

14% 

0% 

Table  E-22.  Percent  by  Pilot  Reporting  Post-Training  Perceptions  of  Management’s  Projections 
of  Likelihood  of  Successful  IPPD  Application 


Values  Pilots 


1 

2 

3 

4 

5 

6 

7 

8 

9 

10 

Don’t  Know 

22% 

50% 

28% 

40% 

100% 

14% 

14% 

14% 

57% 

0% 

Highly  Unlikely  to 
Unlikely 

22% 

8% 

21% 

0% 

0% 

0% 

0% 

86% 

29% 

0% 

50% 

34% 

21% 

34% 

60% 

0% 

29% 

57% 

0% 

14% 

67% 

Likely  to  Highly 
Likely 

22% 

21% 

17% 

0% 

0% 

57% 

29% 

0% 

0% 

33% 

Table  E-23.  Percent  by  Pilot  Reporting  Post-Training  Perceptions  of  Management’s  Projections 
About  Achieving  Benefits  from  IPPD 


Values  Pilots 


1 

2 

3 

4 

5 

6 

7 

8 

9 

10 

Don’t  Know 

13% 

50% 

28% 

40% 

100% 

29% 

14% 

43% 

57% 

0% 

Highly  Unlikely  to 
Unlikely 

25% 

13% 

21% 

0% 

0% 

0% 

0% 

57% 

29% 

0% 

50% 

37% 

17% 

44% 

60% 

0% 

28% 

43% 

0% 

14% 

67% 

Likely  to  Highly 
Likely 

25% 

21% 

7% 

0% 

0% 

43% 

43% 

0% 

0% 

33% 
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Table  E-24.  Percent  by  Pilot  Reporting  Post-Training  Perceptions  of  Management’s  Support  of 
IPPD  Attendance 


Values 

1 

2 

3 

4 

Pilots 

5  6 

7 

8 

9 

10 

Don’t  Know 

0% 

4% 

10% 

20% 

0% 

0% 

0% 

0% 

0% 

0% 

No  to  Slight 

22% 

17% 

7% 

20% 

0% 

0% 

14% 

14% 

14% 

0% 

Moderate 

11% 

12% 

24% 

0% 

100% 

14% 

0% 

15% 

43% 

0% 

Strong  to  Very 
Strong 

67% 

67% 

59% 

60% 

0% 

86% 

86% 

71% 

43% 

100% 

Table  E-25.  Percent  by  Pilot  Reporting  Amount  of  Knowledge  of  EPPD  Concepts,  Months  After 
Training 


Values 

Pilots 

1 

2 

3 

4 

5 

6 

7 

8 

9 

10 

11 

None  to  Slight 

33% 

18% 

33% 

6% 

36% 

0% 

0% 

11% 

17% 

20% 

0% 

Moderate 

50% 

76% 

67% 

50% 

46% 

67% 

40% 

22% 

67% 

40% 

100% 

Large  to  Very  Large 

17% 

6% 

0% 

44% 

18% 

33% 

60% 

67% 

16% 

40% 

0% 

Table  E-26.  Percent  by  Pilot  Reporting  Levels  of  Understanding  of  IPPD  Initiative,  Months  After 
Training 


Values 

1 

2 

3 

4 

5 

Pilots 

6 

7 

8 

9 

10 

11 

None  to  Slight 

17% 

6% 

0% 

6% 

27% 

0% 

0% 

11% 

0% 

0% 

0% 

Moderate 

16% 

59% 

33% 

12% 

18% 

33% 

0% 

0% 

33% 

0% 

0% 

Good  to  Excellent 

67% 

35% 

67% 

82% 

55% 

67% 

100% 

89% 

67% 

100% 

100% 

Table  E-27.  Percent  by  Pilot  Reporting  Levels  of  Understanding  of  EPPD  Benefits,  Months  After 
Training 


Values 

1 

2 

3 

4 

5 

Pilots 

6 

7 

8 

9 

10 

11 

None  to  Slight 

33% 

12% 

33% 

0% 

27% 

0% 

0% 

11% 

0% 

0% 

0% 

Moderate 

17% 

47% 

33% 

19% 

36% 

33% 

40% 

78% 

33% 

20% 

0% 

Good  to  Excellent 

50% 

41% 

34% 

81% 

37% 

67% 

60% 

11% 

67% 

80% 

100% 

Table  E-28.  Percent  by  Pilot  Reporting  Levels  of  Understanding  of  IPPD  Application,  Months 
After  Training 


Values 

1 

2 

3 

4 

5 

Pilots 

6 

7 

8 

9 

10 

11 

None  to  Slight 

50% 

23% 

67% 

6% 

36% 

33% 

0% 

0% 

83% 

40% 

0% 

Moderate 

33% 

65% 

33% 

44% 

37% 

67% 

60% 

22% 

0% 

0% 

100% 

Good  to  Excellent 

17% 

12% 

0% 

50% 

27% 

0% 

40% 

78% 

17% 

60% 

0% 

Table  E-29.  Percent  by  Pilot  Reporting  Extent  Capable  of  Applying  DPPD,  Months  After  Training 


Values 

Pilots 

1 

2 

3 

4 

5 

6 

7 

8 

9 

10 

11 

Not  to  Slightly 

50% 

47% 

67% 

25% 

46% 

33% 

0% 

11% 

33% 

20% 

25% 

Moderately 

33% 

47% 

33% 

50% 

36% 

0% 

40% 

11% 

50% 

40% 

50% 

Capable  to  Very 
Capable 

17% 

6% 

0% 

25% 

18% 

67% 

40% 

78% 

17% 

60% 

25% 

Table  E-30.  Percent  by  Pilot  Reporting  Additional  IPPD  Training  or  Mentoring  Needed,  Months 
After  Training _ 


Values 

1 

2 

3 

4 

5 

Pilots 

6 

7 

8 

9 

10 

11 

None  to  Slight 

33% 

12% 

0% 

25% 

30% 

67% 

80% 

44% 

33% 

80% 

25% 

Moderate 

33% 

53% 

67% 

50% 

60% 

33% 

20% 

56% 

67% 

10% 

75% 

Large  to  Very  Large 

34% 

35% 

33% 

25% 

10% 

0% 

0% 

0% 

0% 

0% 

0% 
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Table  £-31.  Number  by  Pilot  Reporting  Various  Amounts  of  Mentoring  Communications 


Number  Of 
Communications 

Pilots 

Reported 

1 

2 

3 

4 

5 

6 

7 

8 

9 

10 

11 

0 

6 

5 

2 

14 

7 

1 

2 

7 

5 

2 

3 

1 

2 

1 

3 

2 

2 

2 

1 

3 

3 

1 

1 

2 

4 

2 

1 

1 

6 

2 

1 

2 

8 

1 

10 

1 

16 

1 

20 

1 

Table  E-32.  Percent  by  Pilot  Reporting  Management’s  Knowledge  of  IPPD  Concepts  and 
Methods,  Months  After  Training 


Values 

1 

2 

3 

4 

5 

Pilots 

6 

7 

8 

9 

10 

11 

Don’t  Know 

0% 

6% 

33% 

05 

0% 

0% 

20% 

0% 

50% 

0% 

0% 

None  to  Small 

83% 

53% 

33% 

56% 

56% 

67% 

0% 

33% 

33% 

60% 

50% 

Moderate 

17% 

29% 

34% 

6% 

36% 

33% 

60% 

56% 

17% 

40% 

25% 

Large  to  Very  Large 

0% 

12% 

0% 

37% 

9% 

0% 

20% 

11% 

0% 

0% 

25% 

Table  E-33.  Percent  by  Pilot  Reporting  Management’s  Understanding  of  How  IPPD  Methods 
Apply  to  Projects,  Months  After  Training 


Values 

1 

2 

3 

4 

5 

Pilots 

6 

7 

8 

9 

10 

11 

Don’t  Know 

05 

6% 

.0% 

6% 

0% 

0% 

20% 

22% 

50% 

0% 

0% 

None  to  Slight 

83% 

65% 

100% 

56% 

45% 

100% 

20% 

22% 

33% 

40% 

25% 

Moderate 

17% 

23% 

0% 

25% 

46% 

0% 

20% 

45% 

0% 

20% 

25% 

Good  to  Excellent 

0% 

6% 

0% 

13% 

9% 

0% 

40% 

11% 

17% 

40% 

60% 
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Table  E-34.  Percent  by  Pilot  Reporting  Management’s  Levels  of  Success  of  IPPD  Application, 
Months  After  Training _ 


Values 

1 

2 

3 

4 

5 

Pilots 

6 

7 

8 

9 

10 

11 

Don’t  Know 

17% 

23% 

0% 

19% 

27% 

33% 

40% 

33% 

67% 

20% 

25% 

Weak  to 

Nonexistent 

83% 

41% 

67% 

48% 

27% 

0% 

20% 

11% 

33% 

60% 

75% 

Mediocre 

0% 

29% 

33% 

18% 

28% 

33% 

0% 

22% 

0% 

20% 

0% 

Successful  to  Highly 
Successful 

0% 

7% 

0% 

18% 

19% 

34% 

40% 

34% 

0% 

0% 

0% 

Table  E-35.  Percent  by  Pilot  Reporting  Management’s  Extent  of  Benefits  from  IPPD,  Months 
After  Training 


Values 

Pilots 

1 

2 

3 

4 

5 

6 

7 

8 

9 

10 

11 

Don’t  Know 

16% 

35% 

0% 

19% 

27% 

0% 

60% 

44% 

50% 

20% 

25% 

None  to  Small 

67% 

65% 

66% 

50% 

46% 

66% 

20% 

0% 

50% 

60% 

75% 

Moderate 

17% 

0% 

0% 

19% 

27% 

34% 

20% 

44% 

0% 

20% 

0% 

Large  to  Very  Large 

0% 

0% 

34% 

12% 

0% 

0% 

0% 

12% 

0% 

0% 

0% 

Table  E-36.  Percent  by  Pilot  Reporting  Management’s  Support  for  Initial  IPPD  Training  Months 
After  Training 


Values 

Pilots 

1 

2 

3 

4 

5 

6 

7 

8 

9 

10 

11 

Don’t  Know 

16% 

0% 

0% 

6% 

0% 

0% 

0% 

0% 

16% 

0% 

0% 

None  to  Slight 

16% 

6% 

0% 

25% 

0% 

0% 

0% 

0% 

17% 

0% 

0% 

Moderate 

17% 

35% 

33% 

57% 

0% 

0% 

20% 

80% 

22% 

50% 

20% 

75% 

Strong  to  Very 
Strong 

50% 

59% 

67% 

12% 

100% 

100% 

78% 

78% 

17% 

80% 

25% 
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Table  E-37.  Percent  by  Pilot  Reporting  Management’s  Support  for  Supplemental  Training  or 
Mentoring  Use,  Months  After  Training 


Values 

1 

2 

3 

4 

5 

Pilots 

6 

7 

8 

9 

10 

11 

Don’t  Know 

17% 

0% 

0% 

19% 

20% 

0% 

0% 

22% 

17% 

0% 

0% 

None  to  Slight 

17% 

37% 

0% 

50% 

50% 

33% 

60% 

0% 

16% 

60% 

50% 

Moderate 

33% 

31% 

67% 

19% 

0% 

33% 

20% 

33% 

50% 

20% 

25% 

Strong  to  Very 
Strong 

33% 

32% 

33% 

12% 

30% 

34% 

20% 

45% 

17% 

20% 

25% 

Table  E-38.  Percent  by  Pilot  Reporting  Management’s  Support  for  Use  of  EPPD  Methods,  Months 
After  Training 


Values 

1 

2 

3 

4 

5 

Pilots 

6 

7 

8 

9 

10 

11 

Don’t  Know 

16% 

6% 

0% 

0% 

9% 

0% 

0% 

11% 

33% 

0% 

0% 

None  to  Slight 

67% 

44% 

33% 

19% 

55% 

0% 

0% 

22% 

33% 

40% 

50% 

Moderate 

0% 

19% 

33% 

50% 

9% 

67% 

60% 

22% 

17% 

40% 

25% 

Strong  to  Very 
Strong 

17% 

31% 

34% 

31% 

27% 

33% 

40% 

55% 

17% 

20% 

25% 

Table  E-39.  Percent  by  Pilot  Reporting  Extent  IPPD  Use  Benefited  Their  Program,  Months  After 
Training 


Values 

1 

2 

3 

4 

5 

Pilots 

6 

7 

8 

9 

10 

11 

None  to  Slightly 

100% 

71% 

100% 

56% 

64% 

33% 

20% 

25% 

100% 

100% 

100% 

Moderately 

0% 

29% 

0% 

25% 

27% 

67% 

60% 

63% 

0% 

0% 

0% 

Greatly  to 
Excellently 

0% 

0% 

0% 

19% 

9% 

0% 

20% 

12% 

0% 

0% 

0% 
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Table  E-40.  Percent  by  Pilot  Reporting  Extent  IPPD  Could  Benefit  Other  6.3  Programs,  Months 
After  Training 


Values 

1 

2 

3 

4 

5 

Pilots 

6 

7 

8 

9 

10 

11 

None  to  Small 

17% 

18% 

67% 

6% 

18% 

0% 

0% 

0% 

0% 

40% 

25% 

Moderate 

50% 

41% 

33% 

44% 

37% 

33% 

60% 

44% 

20% 

40% 

25% 

Large  to  Very  Large 

33% 

41% 

0% 

50% 

45% 

67% 

40% 

56% 

80% 

20% 

50% 

Table  E-41.  Percent  by  Pilot  Reporting  IPPD  Helped  Reduced  Costs,  Months  After  Training 


Values 

1 

2 

3 

4 

5 

Pilots 

6 

7 

8 

9 

10 

11 

Don’t  Know 

17% 

53% 

0% 

25% 

55% 

100% 

80% 

50% 

0% 

0% 

25% 

Not  at  All  to  Slight 

83% 

41% 

100% 

56% 

36% 

0% 

20% 

50% 

100% 

60% 

75% 

Moderate 

0% 

0% 

0% 

6% 

9% 

0% 

0% 

0% 

0% 

40% 

0% 

Large  to  Very  Large 

0% 

6% 

0% 

13% 

0% 

0% 

0% 

0% 

0% 

0% 

0% 

Table  E-42.  Percent  by  Pilot  Reporting  IPPD  Helped  Shorten  Timelines,  Months  After  Training 


Values 

1 

2 

3 

4 

5 

Pilots 

6 

7 

8 

9 

10 

11 

Don’t  Know 

17% 

35% 

33% 

38% 

55% 

100% 

20% 

37% 

0% 

0% 

25% 

Not  at  All  to  Slight 

83% 

59% 

67% 

37% 

36% 

0% 

60% 

63% 

100% 

100% 

75% 

Moderate 

0% 

6% 

0% 

25% 

9% 

0% 

20% 

0% 

0% 

0% 

0% 

Large  to  Very  Large 

0% 

0% 

0% 

0% 

0% 

0% 

0% 

0% 

0% 

0% 

0% 
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Table  E-43.  Percent  by  Pilot  Reporting  IPPD  Helped  Increase  Productivity,  Months  After 
Training 


Values 

1 

2 

3 

4 

5 

Pilots 

6 

7 

8 

9 

10 

11 

Don’t  Know 

17% 

41% 

0% 

12% 

45% 

0% 

0% 

38% 

0% 

0% 

25% 

Not  at  All  to  Slight 

83% 

53% 

100% 

44% 

46% 

0% 

60% 

50% 

100% 

80% 

75% 

Moderate 

0% 

6% 

0% 

25% 

9% 

100% 

40% 

12% 

0% 

20% 

0% 

Large  to  Very  Large 

0% 

0% 

0% 

19% 

0% 

0% 

0% 

0% 

0% 

0% 

0% 

Table  E-44.  Percent  by  Pilot  Reporting  IPPD  Helped  Identify  Risks,  Months  After  Training 


Values 

1 

2 

3 

4 

5 

Pilots 

6 

7 

8 

9 

10 

11 

Don’t  Know 

17% 

23% 

0% 

6% 

36% 

0% 

20% 

12% 

0% 

0% 

25% 

Not  at  All  to  Slight 

66% 

53% 

66% 

50% 

36% 

66% 

20% 

25% 

100% 

60% 

75% 

Moderate 

17% 

24% 

34% 

25% 

28% 

34% 

40% 

63% 

0% 

40% 

0% 

Large  to  Very  Large 

0% 

0% 

0% 

19% 

0% 

0% 

40% 

0% 

0% 

0% 

0% 

Table  E-45.  Percent  by  Pilot  Reporting  IPPD  Course  Useful  for  Professional  Development, 
Months  After  Training 


Values 

1 

2 

3 

4 

5 

Pilots 

6 

7 

8 

9 

10 

11 

No  Use  to  Very 

Little 

17% 

12% 

0% 

6% 

18% 

33% 

20% 

11% 

0% 

20% 

0% 

Moderately  Useful 

68% 

47% 

100% 

56% 

46% 

67% 

40% 

22% 

50% 

20% 

50% 

Quite  to  Extremely 
Useful 

16% 

41% 

0% 

38% 

36% 

0% 

40% 

67% 

50% 

60% 

50% 

83 


Table  E-46.  Percent  by  Pilot  Reporting  Importance  of  All  Government  Members  Attending 
Course 


Values 

1 

2 

3 

4 

5 

Pilots 

6 

7 

8 

9 

10 

11 

Not  Important 

33% 

23% 

0% 

31% 

22% 

0% 

80% 

0% 

17% 

20% 

0% 

Yes,  At  Some  Point 

17% 

12% 

67% 

44% 

67% 

33% 

0% 

67% 

33% 

60% 

50% 

Yes,  At  Same  Time 

50% 

65% 

33% 

25% 

11% 

67% 

20% 

33% 

50% 

20% 

50% 

Table  E-47.  Percent  by  Pilot  Reporting  Importance  of  All  Contractors  Attending  Course 


Values 

1 

2 

3 

4 

5 

Pilots 

6 

7 

8 

9 

10 

11 

Not  Important 

17% 

18% 

0% 

47% 

9% 

0% 

80% 

0% 

17% 

40% 

0% 

Yes,  At  Some  Point 

0% 

35% 

67% 

40% 

45% 

33% 

20% 

56% 

33% 

20% 

50% 

Yes,  At  Same  Time 

83% 

47% 

33% 

13% 

46% 

67% 

0% 

44% 

50% 

40% 

Table  E-48.  Percent  by  Pilot  Recommending  IPPD  Course  for  Colleagues  on  Other  Projects 


Values 

Pilots 

1 

2 

3 

4 

5 

6 

7 

8 

9 

10 

11 

No 

0% 

12% 

0% 

0% 

18% 

0% 

20% 

0% 

17% 

40% 

0% 

Will  Provide  A 
Little  Benefit 

50% 

47% 

33% 

25% 

18% 

0% 

20% 

11% 

16% 

20% 

50% 

Appropriate  For 
Some 

33% 

41% 

67% 

50% 

37% 

100% 

60% 

67% 

67% 

20% 

25% 

Strongly 

17% 

0% 

0% 

25% 

27% 

0% 

0% 

22% 

0% 

20% 

25% 

Recommended 


APPENDIX  F:  SUMMARY  OF  IPPD  APPLICATION  EXPERIENCES 

OF  PILOT  IPTS 


SUMMARY  OF  IPPD  IMPLEMENTATION  IN  THE  PILOT  PROJECTS:  BASED  ON 
TEAM  MEMBERS’  PERCEPTIONS  FROM  FOLLOW-UP  INTERVIEWS 

The  following  pages  summarize  the  researchers’  perceptions  of  the  IPPD  implementation 
experience  among  the  11  pilot  projects  for  which  implementation  was  possible.  These 
summaries  are  based  on  small  group  and  telephone  follow-up  interviews  with  IPT  members. 
Program  managers  have  reviewed  the  summaries  to  correct  erroneous  interpretations  of 
interview  comments  and  to  check  for  accuracy  in  terminology. 


3-DOC  (  3-Dimensional  Optical  Computing  Memory):  This  project  is  attempting  to  develop 
a  computer  memory  storage  device  the  size  of  a  sugar  cube,  capable  of  storing,  retrieving,  and 
writing  data  at  10  to  the  12*h  power. 

a.  Interviewees:  The  researcher  met  with  Bemie  Clark,  Fred  Haritatos,  Gary  Sunada,  Shay 
Capehart,  Ed  Daniszewski,  and  A1  Jamberdino)  in  one  group  to  discuss  their  progress  using 
IPPD  technologies. 

b.  Appropriateness  for  IPPD:  The  group  members  indicated  that  their  project  was  in  too 
much  of  an  early  conceptual  stage  to  employ  the  IPPD  procedures.  They  suggested  that  the 
process  would  be  more  appropriate  at  a  later  stage. 

c.  IPPD  Organization/Activities:  The  group  has  used  few  of  the  IPPD  procedures.  They 
determined  that  IPPD  procedures  did  not  fit  their  idea  of  how  to  approach  the  3-DOC  research 
project  at  this  early  stage  of  development.  They  said  that  many  of  their  team  members  have  been 
engaged  in  this  type  of  research  for  20  years  and  could  make  the  early  decisions  based  on  their 
intuitive  understanding  and  experience.  They  said  that  they  were  using  a  common  sense 
approach  that  IPPD  utilizes  (working  with  customers  and  evaluating  different  alternatives  in 
terms  of  performance  and  cost),  but  they  didn’t  see  the  utility  of  documenting  their  activities  in 
the  detailed  manner  that  IPPD  demands.  In  addition  they  claim  there  are  no  alternative  suppliers 
for  some  of  the  technologies  they’re  working  with,  so  there  is  less  demand  for  testing  alternative 
approaches.  They  felt  the  attempt  to  adapt  these  procedures  to  their  project  would  require 
additional  time  and  resources  which  had  been  in  increasingly  short  supply  because  of  a  sizeable 
loss  of  some  of  their  funding.  The  team  members  indicate  they  have  good  internal 
communication  because  their  offices  are  located  together,  and  they  say  they  have  frequent 
interaction  with  ACC  and  AIA  representatives  who  will  be  their  customers. 

d.  Measuring  Success:  They  have  not  established  IPPD  measures  of  success.  Their  goal  is 
to  provide  an  effective  product  for  their  customers  at  a  reasonable  cost.  They  would  like  to 
achieve  10  to  the  12*h  data  storage  and  retrieval,  but  indicated  that  customers  will  be  happy  to 
get  whatever  they  can  produce. 

e.  Observations:  The  group  indicated  that  part  of  their  motivation  for  volunteering  for 
IPPD  was  the  possibility  of  securing  additional  funding.  However,  when  they  found  out  what 
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IPPD  really  was  and  the  difficulty  of  applying  it  at  their  stage,  they  were  less  willing  to 
participate  because  it  could  cause  potential  delays.  Since  the  team  felt  IPPD  implementation 
would  take  time  and  resources  away  from  the  mainstream  research  and  these  costs  would  have  to 
be  absorbed,  they  saw  it  would  actually  have  an  opposite  effect  on  their  project.  Thus,  IPPD 
procedures  were  not  employed. 

ALEP  (Advanced  Laser  Eye  Protection):  This  IPT  is  working  an  area  of  research  to  improve 
laser  eye  protection  (LEP)  for  aircrew  members.  Their  job  is  to  examine  various  dyes  and 
technologies  that  will  enable  flight  crews  to  perform  all  aspects  of  their  flight  mission  and  still 
protect  them  against  laser  effects.  Their  focus  has  been  on  the  near-term  approaches  to  laser  eye 
protection,  involving  fixed  wavelength  materials  and  technologies. 

a.  Interviewees:  The  researcher  met  with  project  representatives  at  two  locations  over  four 

sessions. 

b.  Appropriateness  for  IPPD:  As  the  laser  eyewear  protection  project  is  one  of  the  most 
6.3-oriented  programs  in  the  Air  Force  Research  Laboratory,  Materials  and  Manufacturing 
Directorate  (AFRL/ML),  it  was  a  better  candidate  for  IPT  selection.  It  also  offered  a  variety  of 
alternative  requirements  and  potential  approaches  that  could  be  considered  using  IPPD 
procedures.  However,  as  it  has  been  an  ongoing  program  for  several  years,  it  may  have  been 
more  difficult  to  introduce  a  new  research  approach  into  a  team  that  has  been  doing  business  its 
standard  way  for  many  years.  Despite  this  potential  obstacle,  the  Program  Manager  indicates 
that  their  IPT  members  seemed  relatively  willing  to  adapt. 

c.  IPPD  Organization/Activities:  This  project  is  divided  among  two  separate  lab  groups: 
the  Hardened  Materials  Branch  at  AFRL/ML  at  Wright  Patterson  AFB,  and  the  Human 
Effectiveness  Directorate  (AFRL/HED)  at  Brooks  AFB.  Primarily  the  ML  group  coordinates 
with  contractors  to  produce  the  eyewear  products,  and  the  HED  group,  including  TASC 
contractors,  carries  out  the  physical,  psychophysical,  and  human  factors  testing  of  the  product. 
This  division  of  labor  clearly  requires  greater  coordination  and  understanding  of  each  group’s 
perspective  in  order  to  have  an  effective  consolidated  effort.  In  short,  the  HED  group  sees  itself 
as  more  near-term,  product  evaluation-oriented,  emphasizing  produceability,  cost  goals,  and 
manufacturability.  The  ML  group,  which  has  been  involved  with  and  supported  production- 
oriented  activities  such  as  tooling,  scale-up,  and  development  of  product  evaluation 
methodologies,  is  still  somewhat  more  involved  with  answering  some  long-term  technology 
questions.  A  good  deal  of  time  has  been  devoted  to  developing  a  common  vision  of  objectives 
and  roles  among  members  of  the  Brooks  and  Wright-Patterson  team. 

One  organizational  difficulty  in  this  project  was  the  limited  System  Program  Office  (SPO) 
involvement  because  the  Life  Support  SPO  did  not  have  anyone  technically  competent  to  cover 
laser  eyewear.  Now  the  TASC  representatives  are  facilitating  the  user  interface  to  determine 
requirements.  This  becomes  compounded  because  there  are  a  large  number  of  potential 
customers  from  different  operational  missions,  aircraft  types,  and  cockpit  environments.  Their 
varying  needs  make  it  more  difficult  to  fix  on  a  set  of  performance  requirements  and  factors  or 
conditions  to  consider  for  testing.  Representatives  from  a  SPO  and  using  commands  were  not 
present  at  the  development  of  an  initial  listing  of  potential  program  objectives  that  occurred  just 
after  the  IPPD  training  course  in  May  97.  Nor  were  they  present  at  the  earliest  HOQ 
development  sessions,  as  the  project  staff  felt  having  them  involved  in  detailed  negotiations  of 
performance  requirements  would  have  been  a  poor  use  of  the  users’  time.  The  researchers 
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indicate  that  the  Operational  Requirements  Documents  (ORD)  and  the  LEP  body  of  work  over 
the  years  have  provided  enough  user  feedback  to  have  a  good  idea  of  the  users’  perspective. 

The  major  contractors  (Kaiser  and  Rockwell)  were  also  not  involved  much  in  selecting  the 
project  requirements  or  initial  building  of  the  HOQ.  However,  some  of  their  representatives  had 
attended  the  IPPD  and  DOE  training  sessions.  Project  members  from  ML  indicate  they  spent 
considerable  time  keeping  users  informed  at  tactics  conferences  and  providing  technology 
demonstrations  and  briefings  at  using  commands.  The  research  staff  plans  to  involve  contractors 
and  customers  more  in  the  next  IPPD  stages  as  they  finalize  the  HOQ  and  determine  the  extent  to 
which  they  will  be  able  to  test  the  potential  technologies. 

By  November,  the  ML  group  had  met  as  an  IPT  on  five  occasions  and  in  joint  sessions  with 
the  AL  group  four  times,  including  a  kick-off  meeting  and  a  planning  session  on  the  day  after 
IPPD  training.  They  are  planning  to  have  an  IPPD  refresher  session  with  all  team  members  and 
then  continue  to  refine  the  HOQ.  They  have  not  yet  implemented  the  Technology  Worksheets 
and  Value  Scorecard  procedures.  A  number  of  tests  of  technology  will  have  to  be  developed  by 
the  team  and  conducted  by  AL  to  get  answers  on  various  technology  options.  It  is  not  clear 
whether  the  contractors  will  be  able  to  devote  the  time  and  resources  needed  to  test  all  those 
options  so  they  can  fully  employ  IPPD  documentation. 

d.  Measuring  Success:  Both  groups  indicated  the  requirements  for  assessing  the  success  of 
IPPD  efforts  were  initially  not  very  clear.  The  AF  desired  outcomes  have  been  made  somewhat 
clearer  as  TARA  presentation  requirements  have  been  made  known.  Wright  Laboratory  (WL) 
staff  feel  they  will  need  to  demonstrate  the  use  of  specific  IPPD  procedures. 

e.  Observations:  Team  members  indicate  that  participation  in  the  IPPD  initiative  has  had 
some  positive  benefits,  primarily  in  bringing  the  two  groups  together  to  decide  on  a  common 
perspective.  The  laser  protection  requirements  profiles,  means  for  quantifying  those 
requirements,  and  exit  criteria  have  been  established  and  would  have  been  accomplished  without 
IPPD  procedures.  However,  the  PM  suggests  that  the  IPPD  methodology  provided  a  more 
structured  approach  to  accomplishing  these  objectives  and  also  provided  a  valuable 
documentation  trail. 

There  is  also  a  belief,  among  most  team  members,  that  the  use  of  IPPD  procedures  (as  well 
as  being  a  pilot  team)  has  come  with  a  large  amount  of  overhead  in  terms  of  briefings,  meetings, 
and  documentation.  It  is  still  a  question  of  whether  additional  benefits  that  may  be  obtained 
from  the  structured  value  analysis  will  offset  the  costs  of  detailed  data  collection  and  extensive 
testing  that  could  be  required  for  a  full  implementation  of  IPPD. 

Many  of  the  costs  associated  with  IPPD  implementation,  at  least  in  this  pilot  phase,  have  had 
to  come  out  of  the  project  funding.  The  desire  to  fully  implement  the  IPPD  process,  with  all 
potential  tests,  bumps  up  against  the  reality  of  no  additional  funds.  In  the  future,  as  the  IPPD 
learning  curve  is  smaller,  these  additional  costs  may  be  less,  and  they  will  be  built  into  the 
original  project  funding  estimates. 

AMD  (Advanced  Motor  Drive)  This  project  is  developing  an  electromechanical  actuator  (as 
opposed  to  a  hydraulic  system)  with  an  emphasis  on  the  motor  drive  for  a  spoiler  surface  of  a 
transport  aircraft. 
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a.  Interviewees:  The  researcher  met  with  Greg  Fronista  (the  Program  Manager)  and  Earl 
Gregory.  Both  had  taken  the  DSSM  course  at  TILI  in  May  1996.  One  other  team  member  who 
had  participated  in  the  IPPD  training,  a  contractor  from  Sundstrand,  was  not  available  for 
interviewing.  The  PM  indicated  that  there  were  actually  nine  members  on  the  IPT,  but  only 
those  three  had  attended  the  DSSM  or  IPPD  course. 

b.  Appropriateness  for  IPPD:  One  difficulty  associated  with  application  of  IPPD  methods 
in  this  IPT  is  that  the  project  was  already  well  underway  when  it  was  selected  as  an  IPT.  The 
preliminary  design  was  already  complete,  and  team  members  actually  looked  back  at  previous 
decisions  to  determine  if  appropriate  decisions  had  been  made.  Greg  feels  the  project  was 
selected  because  of  its  complexity  and  because  the  contractor,  Sundstrand,  had  a  strong 
manufacturing  base  and  some  low-level  experience  with  basic  IPPD  principles. 

c.  IPPD  Organization/Activities:  Greg  was  the  main  government  player,  and  he  met  with 
four  contractor  representatives  at  Sundstrand  every  two  weeks  for  several  months  to  determine 
how  they  would  employ  IPPD  processes.  Meetings  included  all  IPPD  players,  but  not  all  of  the 
IPPD  team  members  were  IPPD  trained.  James  Gregory  trainers  were  at  all  these  Sundstrand 
sessions.  One  TI  mentor  also  sat  in  on  a  DOE  session  at  Sundstrand.  They  also  communicated 
through  phone,  fax,  and  mail,  but  felt  face-to-face  sessions  kept  key  players  well  connected. 

During  these  meetings  the  IPT  developed  an  HOQ,  performed  a  designed  experiment 
(Taguchi  3-factor,  2-level  design)  to  evaluate  a  new  approach  to  concurrent  design  during  S&T. 
Data  was  generated  using  an  EMA  simulation  application  at  Sundstrand.  A  model  was 
developed  relating  deployment  rate  for  a  large  spoiler  surface.  In  addition,  a  tolerance  analysis 
was  done  for  the  Phase  II  controller  electronics,  using  a  tool  developed  for  the  S&T  IPPD 
process  and  distributed  during  the  AT/IPPD  course.  Sundstrand  engineers  were  enthusiastic 
about  using  that  tool. 

Implementation  of  value  analysis  aspects  of  the  IPPD  process  was  more  difficult.  The 
project  was  far  enough  along  that  the  IPT  had  to  recreate  the  rationale  for  their  decisions.  They 
had  done  some  simulations  of  motor  design  options  to  test  those  options,  but  the  choice  of  motor 
type  (switch  reluctance)  was  made  because  of  its  fault  tolerance  and  cost  considerations.  The 
IPT  used  similar  existing  knowledge  or  general  data  to  compare  performance,  producibility,  and 
cost  tradeoffs,  but  the  decisions  were  not  documented  using  technology  worksheets  or  a  value 
scorecard. 

d.  Measuring  success:  Greg  could  name  no  specific  criteria  forjudging  “success”  of  IPPD 
efforts.  They  felt  the  primary  concern  was  to  show  they  were  making  an  effort  to  employ  IPPD 
procedures 

e.  Observations:  Having  to  apply  IPPD  procedures  retrospectively  to  already  completed 
decisions  was  like  running  two  programs  and  doubled  some  of  the  workload  on  this  project. 
They  discovered  that  many  of  the  considerations  identified  by  the  IPPD  process  had  been 
informally  considered  and  accomplished  originally,  but  they  were  not  recognized  and 
documented  as  part  of  a  formal  affordability  determination  process 

-  It  is  important  to  have  a  critical  mass  of  team  members  knowledgeable  enough  about  IPPD 
to  come  to  some  decisions  about  its  use.  On  some  decisions  the  contractor  was  IPPD- 
knowledgeable  and  the  activity  was  right  for  employment  of  the  IPPD  process,  but  the  rest  of  the 
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actors  at  the  meeting  weren’t  as  knowledgeable,  the  attempt  to  introduce  IPPD  processes  bogged 
down,  and  the  initiative  was  lost.  They  suggest  most  team  members  should  have  at  least  a  basic 
level  of  IPPD  training. 

-  It  is  most  important  for  contractors  to  be  IPPD-capable  and  convinced  it  is  worth  the  effort. 
They  are  not  sure  Sundstrand’s  totally  convinced  yet  that  the  process  can  be  applied  to  6.3 
programs.  Demand  for  affordability  measures  should  be  written  into  contracts  so  capability  in 
that  area  and  incentive  to  demonstrate  employment  of  IPPD  procedures  are  built  into  the 
selection  process. 

-  They  felt  that,  with  the  right  combination  of  trained  team  members  and  the  mentorship 
provided  by  the  training  team,  they  could  have  implemented  the  process  successfully  from  the 
beginning  of  the  project. 

-  They  were  most  comfortable  with  building  the  House  of  Quality  and  thought  that  procedure 
ought  to  be  applied  to  most  6.3  programs  to  help  members  thoughtfully  lay  out  requirements. 

-  In  some  other  decisions,  the  various  players’  competing  interests  related  to  affordability  made 
it  difficult  to  reach  agreements  (e.g.,  SPOs  want  return  on  investment  over  a  few  years,  providing 
more  incentive  to  hold  down  acquisition  costs  at  expense  of  life-cycle  costs). 

-  They  suggested  that  collection  of  data  in  some  projects  can  be  done  through  computer 
simulation  and  tools  for  that  process  need  to  be  developed  or  made  more  available. 

-  Sundstrand  had  indicated  that  a  rough-cut  version  of  the  IPPD  cost  vs  risks  and 
manufacturing  vs  maintaining  considerations  (available  via  an  automated  program)  would  help 
them  justify  programs  and  get  them  off  the  ground. 

CAI  (Composites  Affordability  Initiative):  The  CAI  project  consists  of  three  associated 
programs  designed  to  achieve  a  significant  reduction  in  acquisition  costs  of  aircraft  structures 
through  revolutionary  utilization  of  composite  materials.  Two  near-term  programs  (Tech 
Transition  1  and  Tech  Transition  2)  focus  on  finding  improvements  for  a  specific,  next 
generation  fighter.  This  focus  on  near-term  risk  reduction  means  they  have  to  fit  proposed 
innovations  into  existing  fighter  configurations.  A  longer  term  “Pervasive”  program  affords  a 
more  open-ended  review  of  all  assumptions  and  aspects  of  structural-related  materials,  design,  or 
processes  that  could  affect  the  cost  while  maintaining  quality  of  fighter  aircraft  of  the  future.  In 
this  program  researchers  are  freer  to  question  requirements  and  specifications  and  the  usual  ways 
of  doing  business. 

a.  Interviewees:  The  researcher  met  with  five  members  of  the  IPT  at  Wright-Patterson 
AFB:  Dave  Beeler  (PM),  Diana  Carlin,  Dick  Holzwarth,  and  Ken  Ronald  from  AFRL/  ML,  and 
Paul  Hauwiller  (project  support  contractor  from  Anteon).  Most  of  the  29  team  members  who 
attended  IPPD  training  were  from  the  large  group  of  contractors  (Boeing,  Lockheed-Martin, 
Northrop,  McDonnell  Douglas)  and  were  queried  via  the  follow-up  survey.  A  telephone 
interview  was  also  completed  with  Dee  Gill.,  the  Systems  Engineering  Team  lead  for  the 
Pervasive  program. 

b.  Appropriateness  for  IPPD:  This  program  is  somewhat  challenging  as  an  IPPD  IPT 
because  of  the  highly  conceptual  nature  of  its  mission. 


89 


C.  IPPD  Organization/Activities:  This  program  is  more  complex  because  it  is  divided  into 
three  separate  but  affiliated  programs:  T-l,  T-2,  and  Pervasive.  There  is  an  overall  Leadership 
IPT  (LIPT)  overseeing  management,  planning,  and  integration  of  the  three  subprograms.  In  turn, 
each  program  has  an  Activity  IPT  (AIPT).  The  Pervasive  program  is  also  subdivided  according 
to  tasks  or  functional  areas  (e.g.,  concepts,  fabrication,  assembly,  testing,  cost)  with  a  systems 
engineering  team  (representing  major  stakeholders)  reviewing  the  10  task  teams.  In  addition, 
there  are  both  Navy  and  Air  Force  partners,  four  industrial  partners,  and  numerous 
subcontractors. 

Description  of  the  IPPD  status  of  the  T-l  and  T-2  programs  is  somewhat  constrained  by  the 
need  to  protect  proprietary  information  of  the  two  competing  design  teams. 

The  initial  introduction  of  the  IPPD  process  was  presented  to  the  CAI  LIPT  in  May  1997, 
and  that  group  agreed  to  participate  as  an  IPPD  pilot.  The  AT/IPPD  training  was  presented  to  29 
CAI  members  in  mid- August  1997,  and  a  one-day  follow-on  session  with  the  training 
consultants  was  held  to  outline  an  IPPD  implementation  strategy.  By  that  time,  the  first  phase  of 
the  CAI  program  (getting  organized  and  planning  the  structure  and  ground  rules  for  team 
interaction  and  communication)  had  been  ongoing  for  about  a  year.  T-l  and  T-2  held  their 
kickoff  in  the  summer  of  1997.  The  Pervasive  program  was  begun  in  December  1997. 

The  potential  users/customers  include  both  military  and  industry  partners.  Although  the 
project  focus  is  a  military  aircraft  (JSF),  with  requirements  input  from  potential  users  of  that 
aircraft  through  the  JSF  SPO,  commercial/industry  contractors  are  potential  users  of  the 
technology  lessons  derived,  and  are,  therefore,  significant  stakeholders  and  active  IPT  members. 

Although  they  did  not  specifically  employ  a  House  of  Quality  approach,  the  T-l  and  T-2 
teams  have  established  milestones  and  exit  criteria  for  each  technology  area.  They  have  also 
begun  some  preliminary  investigation  of  performance,  cost,  and  risk  alternatives  for  the  various 
technology  options.  In  doing  so  they  have  not  used  the  technology  worksheets  demonstrated  in 
the  IPPD  design  analysis  process.  Rather  they  indicate  they  have  found  it  effective  to  use  other 
multi-attribute  decision  tools  in  implementing  a  more  traditional  trade  studies  approach.  They 
suggest  that  they  are  in  a  conceptual  stage  that  is  less  conducive  to  the  use  of  the  specific  IPPD 
tools  demonstrated  in  the  training.  At  this  stage  of  consideration  of  conceptual  options  there  are 
often  limited  data  pertaining  to  the  actual  performance  or  manufacturability  of  the  technology 
concepts.  They  have  used  these  data  and  a  QFD  approach  to  qualitatively  assess  differences 
between  the  various  innovation  concepts  and  to  support  decisions  that  will  lead  to  down  selection 
to  a  single  design  concept.  Once  they  have  selected  a  single  concept,  they  will  do  more  testing 
of  the  aspects  of  the  design,  collecting  data  to  refine  the  design.  Ultimately,  when  they  have  a 
more  firm  design,  they  will  do  the  kinds  of  testing  of  its  features  for  performance,  risk,  and 
producibility  that  may  be  more  easily  documented  with  the  IPPD  technology  worksheets  and 
value  scorecard. 

The  Pervasive  AIPT  had  a  relatively  broad  mission  of  improving  the  use  of  composites  in 
aircraft  design  by  developing  tools  and  technologies  needed  to  achieve  breakthrough  reductions 
in  cost  and  weight.  Because  they  had  the  ability  to  look  at  all  potential  areas  of  structure  design 
and  manufacturing  tools  and  processes,  and  the  option  to  question  all  requirements/specs,  the 
team  needed  to  determine  an  initial  area  on  which  to  begin  focusing  their  efforts.  Thus,  in 
September,  the  Pervasive  team  began  the  process  of  determining  areas  of  technology  to  focus  on. 
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To  accomplish  this  decision  process  they  employed  a  modified  HOQ,  using  a  subjective  rating 
system  to  compare  potential  technology  investigation  areas  on  several  criteria, ,  such  as 
“pervasiveness”  (utility  across  structural  functions),  impact  on  affordability,  or  risk  for 
technology  maturation.  This  process  proved  to  be  beneficial  in  requiring  proponents  of  various 
candidate  areas  to  demonstrate  how  their  area  met  the  decision  criteria,  and  it  provided  a  means 
of  documenting  their  decision  process.  Both  attributes  demonstrated  the  utility  of  that  type  of 
structured  approach  to  decision  making  that  is  typical  of  the  IPPD  process. 

In  the  Pervasive  project,  IPT  members  have  been  working  closely  with  training  consultants 
to  develop  and  try  out  new  tools  for  implementing  the  IPPD  procedures.  For  example,  they  have 
used  a  Desirability  Factor  metric  to  assist  decision  making  in  the  House  of  Quality.  They  have 
been  meeting  frequently  with  the  consultants  trying  to  determine  ways  to  quantify  performance, 
cost,  and  producibility  requirements.  When  they  selected  the  10  technology  areas  to  focus  on, 
they  discovered  the  existent  data  were  not  as  fully  developed  or  had  not  been  tested  in  as  robust  a 
manner  as  they’d  have  liked.  They  are  currently  doing  trade  studies  using  revised  technology 
worksheets  developed  by  the  James  Gregory  consultants.  Despite  the  fact  that  some  of  the 
Pervasive  team  members  had  IPD  process  experience  in  the  F-18  and  777  production  programs, 
few  had  experience  using  these  procedures  in  an  R&D  environment.  They  are  finding  that  they 
are  having  some  difficulty  applying  the  Six  Sigma  risk  measures  to  their  analyses  6f  design 
options.  They  indicate  that  some  processes  they  need  to  test  are  so  new  that  no  corresponding 
risk  data  exist,  and  it  would  be  very  costly  to  build  a  part  and  do  the  tests  needed  to  gather  data. 
Thus  the  use  of  the  IPPD  process  is  not  a  simple  task,  but  they  are  working  at  it. 

d.  Measuring  Success:  The  T-l  and  T-2  teams,  possibly  because  of  time  constraints,  are 
more  focused  on  the  mission  of  identifying  and  testing  means  for  acquisition  cost  reduction. 
They  are  concerned  with  measuring  their  accomplishment  of  these  goals,  feel  their  current 
processes  are  helping  achieve  them,  and  have  devoted  less  energy  to  measuring  the  extent  they 
have  employed  the  IPPD  procedures  and  to  documenting  their  use.  The  Pervasive  team  has 
been  able  to  go  through  the  steps  of  developing  means  for  employing  the  IPPD  procedures,  and 
they  are  assessing  their  progress  in  the  use  of  the  IPPD  procedures  as  well  as  accomplishment  of 
their  mission.  They  suggest  it  is  too  early  in  their  project  to  gauge  the  success  of  their 
implementation  or  the  benefits  derived. 

e.  Observations:  It  is  obvious  that  the  CAI  project  provides  a  good  demonstration  of  the 
difficulty  in  implementing  IPPD  to  very  conceptual  R&D  projects  and  of  the  divided  opinion 
regarding  its  utility  at  various  stages  of  the  research  process.  The  Pervasive  team  seems  to  have 
both  the  time  and  commitment  to  stay  with  the  IPPD  process  while  the  tools  for  its 
implementation  are  being  worked  out.  They  have  indicated  that  the  AT/IPPD  course,  as 
originally  delivered,  was  full  of  applicable  processes,  but  needed  more  pertinent  examples  and 
tools  that  would  enable  R&D  personnel  to  carry  out  robust  testing  of  alternatives  and  rigorous 
data-gathering  required  of  the  IPPD  process.  They  indicated  that  the  IPPD  process  still  was 
useful  in  forcing  the  IPTs  to  work  together  off  the  same  page,  sharing  insights  and  perspectives, 
and  making  data-based  decisions  rather  than  relying  just  on  intuitive  assessments  of  experienced 
researchers.  They  emphasized  the  requirement  for  training  of  all  players,  for  extensive 
facilitation  by  credible  experts  after  training,  and  for  program  managers  that  believe  in  the 
process  and  that  will  hold  researchers  accountable  for  providing  credible  data. 


ERCM  (Large  Aircraft  Infrared  Countermeasures):  This  project  has  LIFE  as  its  main 
program,  seeking  techniques  to  defeat  the  threat  of  heat-seeking  missiles.  AH  critical  subsystems 
are  being  addressed  by  this  project,  with  the  early  emphasis  on  the  laser  technology. 

a.  Interviewees:  The  researcher  met  with  12  members  of  the  IPT  over  two  different  90- 
minute  sessions.  Those  participating  included  Mark  Wunderlich  (Program  Manager),  Darrin 
Coffman  (Field  Test  Director),  Bill  Taylor,  (Affordability  Lead),  Rich  Hunziker  (EO  Division 
Chief),  Joe  Koesters  (Branch  Chief),  Virginia  McMillian,  Cal  Verity,  and  Jerry  Dague,  William 
Von  Gunten,  Stanley  Herr,  Adam  Coleman,  and  Harold  Watson  from  the  SAIC  contractors. 

b-  Appropriateness  for  IPPD:  Participants  suggested  this  project  was  amenable  to  IPPD 
because  they  know  who  the  users  are  and  have  contact  with  them.  The  project  had  been 
seriously  underway  for  six  months  prior  to  attendance  of  DSSM  course  in  Oct  1996. 

c-  IPPD  Organization/Activities:  This  process  is  relatively  new  to  members  of  this  project, 
and,  except  for  Litton,  laser  contractors  had  also  had  little  familiarity  with  this  type  of  research 
process.  Some  contractors,  (e.g.,  Lockheed-Martin  and  Sanders)  attended  DSSM  and  DOE 
training,  and  a  number  of  team  members  will  be  attending  the  newer  version  of  the  IPPD  course. 
Implementation  activities  for  IPPD  seem  to  be  concentrated  among  certain  IPT  members.  There 
had  been  several  meetings  called  to  review  progress,  but  they  weren’t  scheduled  on  a  periodic 
basis  or  with  a  lot  of  lead  time,  and  several  members  would  miss  them.  Most  of  the  laser  aspects 
are  being  handled  by  the  PM  in  conjunction  with  the  Directed  Energy  Directorate  and  Lockheed 
contractors.  AFRL/SN’s  representatives  from  the  Laser  Applications  Branch  attended  the  DSSM 
training  but  have  had  little  involvement  with  the  project.  However,  a  key  representative  from 
Simulation  and  Modeling  has  not  attended  the  training.  Some  participants  suggested  the  number 
of  members  actively  involved  has  been  limited  to  save  time  and  resources. 

Right  after  the  DSSM  course  in  late  1996,  a  small  group  put  the  House  of  Quality  together 
for  the  laser  subsystem,  relating  customer  requirements  to  technical  requirements.  This  was  done 
with  the  help  of  James  Gregory  consultants  and  meeting  participants  felt  the  effort  was  very 
beneficial.  In  fact,  these  requirements  had  been  identified  months  earlier  through  strong 
customer  interaction  at  a  Systems  Requirements  Review.  By  the  time  the  DSSM  training  had 
been  provided,  exit  criteria  had  been  established  and  the  IPT  had  already  gone  to  a  Preliminary 
Design  Review.  A  Technology  Transition  Plan  (TTP)  has  been  developed  and  reviewed  with 
representatives  of  the  C-17  SPO  and  Air  Mobility  Command. 

However,  further  movement  toward  development  of  technology  worksheets  and  a  value 
scorecard  and  gathering  the  data  for  those  decisionmaking  tools  has  been  slow.  Affordability 
tradeoff  metrics  need  to  be  established  for  the  various  performance  requirements.  The 
interviewees  suggest  the  obstacle  is  not  enough  time  and  resources.  Originally  the  IPT  thought  it 
would  only  be  able  to  implement  IPPD  in  one  subsystem,  but  now  they  have  reworked  the 
contract  to  require  the  contractor  to  use  IPPD  to  address  affordability  across  all  subsystems. 

d.  Measuring  Success:  They  did  not  feel  that  specific  outcomes  from  the  IPPD 

implementation  were  identified  for  the  IPTs.  Therefore,  they  felt  development  of  a  HOQ  and 
value  analysis  tools  would  show  a  good  faith  effort  to  implement  IPPD  procedures. 

e.  Observations:  Participants  who  had  been  actively  involved  with  the  IPPD 

implementation  reflected  positively  about  the  experience,  suggesting  it  would  have  been 
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beneficial  to  employ  the  HOQ  for  the  entire  system  and  then  work  down  to  the  subsystems.  The 
impression  is  that  the  implementation  is  worked  primarily  among  a  few  core  players,  and  there  is 
relatively  little  communication  with/among  others  regarding  IPPD  status.  Those  who  worked  the 
implementation  indicated  it  was  necessary  to  have  the  consultants’  assistance  in  the  HOQ 
development. 

Participants  suggested  it  is  important  to  have  the  contractors’  initial  buy-in  to  the  IPPD  process 
written  into  the  contract.  The  demand  on  resources  given  to  IPPD  documentation  and  data- 
gathering  are  taken  from  the  project,  and  contractors  don’t  have  additional  funds  beyond  what  is 
required  to  be  spent  on  actual  research  work.  Although  there  was  initial  uncertainty  as  to  the 
overall  payoff  from  the  IPPD  process,  there  seems  to  be  a  somewhat  greater  commitment  of 
interest  and  corporate  funds  aimed  at  supporting  the  affordability  issues. 

A  few  of  the  participants  suggested  the  process  was  not  appropriate  to  6.3  work  .  While 
most  indicated  it  seemed  appropriate,  they  admitted  that  they  were  still  struggling  with  how 
exactly  to  apply  it.  However,  most  participants  saw  the  potential  general  benefit  from  learning 
to  look  at  the  project  beyond  their  own  narrow,  proof-of-concept  and  let-it-go  perspective.  They 
said  it  forces  them  to  think  about  what  happens  later  and  about  potential  ramifications  to 
producibility  and  later  stages. 

Those  who  had  attended  DOE  training  were  very  supportive  of  its  benefits,  suggesting  it 
“should  be  a  standard  course  for  anyone  coming  to  the  lab.”  Others  agreed  but  suggested  the 
DOE  information  should  be  linked  better  to  costs  and  affordability. 

It  was  suggested  that  MANTECH  participation  should  be  funded  earlier  so  they  can  work 
more  as  an  integrated  team 

ISCP  (Improved  Space  Computer  Program):  The  ISCP  project  is  actually  part  of  a  two-phase 
program  to  design  computer  architecture  for  a  space-based  computer  system.  The  first  phase, 
which  kicked  off  in  summer  of  1996,  was  a  contents  analysis  and  definition  phase.  Two 
contractor  teams  did  a  space  systems  requirements  analysis  and  computer  architecture  concept 
definition,  looking  at  three  DOD  missions  and  surveying  industry  to  determine  the  processing 
performance  requirements  for  satellite  systems  in  the  near  future.  They  looked  at  commercial 
available  computer  architecture  that  could  be  employed  in  one  or  more  space  systems,  serving 
both  low-end  and  high-end  missions.  Phase  II,  which  is  known  as  the  Improved  Space 
Architecture  Concept,  is  designed  to  narrow  down  the  potential  architecture  options  to  a 
recommended  design.  Although  the  Phase  II  RFP  went  out  in  the  summer  of  1997,  a  protest  of 
the  bidding  process  delayed  the  final  awarding  of  the  contract  until  April  1998. 

a.  Interviewees:  The  researcher  conducted  telephone  interviews  with  the  IPT  program 
managers  at  Phillips  Lab,  Capt  Joe  Nedeau  and  his  successor,  Mr.  Ken  Hunt. 

b.  Appropriateness  for  IPPD:  The  PM  indicated  the  first  phase  of  the  project,  which  was 
more  focused  on  requirements  definition,  was  not  as  appropriate  for  IPPD  as  the  second  phase,  in 
which  alternative  architectures  will  be  examined  to  determine  the  most  appropriate  for  meeting  a 
variety  of  space  system  missions  at  reasonable  cost. 

c.  IPPD  Organization/  Activities:  Basically,  the  PMs  indicated  the  IPPD  procedures  are  to 
be  employed  much  more  in  Phase  II  than  they  were  in  Phase  I.  In  the  first  phase,  the  potential 
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using  agencies  were  not  as  heavily  involved  as  they  might  have  been.  The  IPT  had  to  project 
where  core  technologies  and  current  systems  are  going,  and,  because  this  is  more  of  a  "tech 
push"  program,  the  users  had  to  be  led  to  consider  the  potential  uses  of  a  space-based  computer 
system.  The  requirements  are  aligned  to  Space  Command  deficiencies,  but,  in  hindsight,  the  PM 
would  have  more  actively  sought  the  involvement  of  the  Space  and  Missile  Systems  Center.  The 
IPPD  requirements  were  not  in  the  first  phase  contract.  Although  the  prime  contractors  attended 
DSSM  training,  few  subs' attended,  and  they  indicated  there  would  be  a  considerable  overhead 
associated  with  IPPD  implementation.  Both  teams  developed  a  common  set  of  customer 
performance  requirements,  and  one  team  attempted  to  quantify  their  expected  architecture 
performance.  However,  most  decisions  were  intuitive,  based  on  expertise  and  experience,  and 
little  other  quantitative  analysis,  testing,  or  documentation  using  design  worksheets  was 
employed. 

In  the  second  phase,  the  IPPD  requirements  are  clearly  outlined  in  the  contract,  and  training 
is  to  be  paid  for  by  AFRL.  The  kickoff  is  set  for  late  May  1998,  and  four  IPTs  will  oversee 
various  aspects  of  the  project.  A  web-site  will  be  set  up  to  connect  contractor  entities.  The  PM 
indicates  the  team  will  do  spreadsheet  analyses  to  filter  out  the  least  desirable  architecture 
alternatives.  They  will  use  initial,  low-level  simulations  and  then  more  detailed,  high-level 
simulations  to  narrow  the  final  four  alternatives  to  a  best  choice.  They  intend  to  use  technology 
worksheets  and  value  scorecards  in  their  documentation  of  decisions  and  indicate  they  will  also 
develop  a  technology  transition  business  plan  (TTBP)  for  each  customer.  Their  intention  is  to 
have  a  software  testbed  capable  of  demonstrating  the  architecture's  capability  for  processing. 

d.  Measuring  Success:  In  the  first  phase,  IPPD  success  was  not  emphasized,  whereas  in 
Phase  II,  use  of  IPPD  procedures  will  be  monitored. 

e.  Observations:  It  seems  that  this  pilot  team  was  not  really  prepared  for  IPPD 
employment  in  the  first  phase  of  the  project,  and  had  less  IPPD  commitment  from  one  of  its  two 
competing  contractors  and  several  subs.  The  PM  stressed  the  need  to  specify  IPPD  requirements 
in  the  contract,  and  he  also  indicated  the  need  for  additional  guidance  in  outlining  these  factors 
contractually.  In  fact,  he  suggested  that  an  S&T  IPPD  set  of  guidelines  be  developed  for 
managers  and  key  contracting  leaders  to  help  them  administer  and  assess  IPPD  implementation. 
The  planned  improvements  in  the  second  phase  indicate  that  the  project  personnel  used  the  first 
phase  as  a  learning  experience. 

MASS  (Modular  Aircraft  Support  System):  This  program  is  designed  to  develop  an  improved 
aerospace  ground  equipment  (AGE)  package  that  has  a  smaller  footprint  for  deployment  (is  more 
reliable  and  maintainable  and  can  be  used  across  multiple  aircraft  types  so  less  equipment  is 
needed).  Also  a  need  for  this  improvement  to  be  affordable  is  emphasized.. 

a.  Interviewees:  The  researcher  met  with  Matt  Tracy  (PM),  Capt  Dwight  Pavek,  and  lLt 
John  Schroeder,  and  had  a  telephone  interview  with  David  Hablanian  (A.D.  Little  contractor). 

b.  Appropriateness  for  IPPD:  Because  the  project  was  begun  in  conjunction  with  the  IPPD 
initiative,  IPPD  supportive  language  was  built  into  the  contract  and  the  contractor  selected  had 
some  IPPD  experience.  In  addition,  the  customers  were  well  known  and  could  help  in 
establishing  user  requirements. 
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c.  IPPD  Organization/Activities:  Fairly  extensive  use  of  integrated  product  teams  with 
good  representation  by  Wright  Lab  personnel,  contractors  (A.D.  Little),  and  potential  customers. 
IPT  meetings  and  a  web  site  have  promoted  good  communication  among  government  managers, 
contractors,  SPOs,  and  users. 

Prior  to  beginning  of  contract,  James  Gregory  consultants  met  twice  with  MASS  IPT, 
including  system  users  to  outline  IPPD  process  and  requirements  and  even  attempted  a  first  cut 
at  a  rough  HOQ.  Source  selection  in  Fall  of  96  included  IPPD  requirements,  and  contractor  reps 
as  well  as  AFRL/HE  personnel  attended  IPPD  training  in  May  97  and  in  September  97.  The 
users,  however  have  not  had  IPPD  training  The  contractors  reviewed  the  rough-cut  HOQ  and 
recommended  modifications.  Multiple  users  (including  Army  and  Navy)  were  very  involved  in 
discussion  of  system  requirements.  This  made  matters  more  difficult  because  of  differing  views 
on  needs  and  trade-off  priorities.  HOQ  was  continually  revised  as  these  different  requirements 
were  identified  or  changed  and  off-the-shelf  parts  and  components  were  discovered.  However, 
10  months  after  training,  the  HOQ  is  fairly  complete,  with  performance  specs  and  technology 
requirements  outlined. 

By  November  1997,  there  had  been  4-5  review  sessions,  but  many  of  the  attendees  did  not 
have  IPPD  training,  and  that  has  hindered  the  progress  that  was  initially  experienced.  At  these 
sessions  members  worked  to  refine  exit  criteria  for  size  and  layout  of  AGE  components. 
Reworking  of  the  HOQ,  reprioritization  of  requirements,  and  many  discussions  regarding 
tradeoffs  based  on  what  technology  was  doable  given  cost,  time,  and  other  requirements  and 
constraints  resulted  in  a  configuration  with  six  concepts.  By  April  1998  the  team  had  identified 
a  single  concept  that  was  actually  a  hybrid  combination  of  the  six. 

In  coming  to  this  decision,  development  of  technology  worksheets  and  a  value  scorecard 
were  appropriate  procedures  as  a  means  of  logically  resolving  some  of  the  tradeoff  debates. 
They  made  an  attempt  at  developing  a  value  scorecard  but  couldn’t  pull  it  off  for  several  reasons: 
they  (a)  didn’t  have  the  technical  ability  to  build  it,  (b)  didn’t  feel  they  had  sufficient  data  or  the 
time  and  resources  to  gather  those  data,  and  (c)  had  difficulty  convincing  senior  enlisted 
members  of  using  agencies  to  accept  this  approach  to  making  these  decisions.  They  weren’t 
ready  to  quantify  each  requirement  and  trade-off  and  were  reluctant  or  unable  to  identify,  collect, 
and  consider  the  data  related  to  life-cycle  costs.  The  IPT  team  members  contend  there  is  no  easy 
system  in  place  for  collecting  data,  such  as  parts  usage  or  failure  rates.  Although  the  contractors 
from  A.D.  Little  did  weight  and  rate  some  of  the  size,  weight,  and  cost  trade-offs,  the  group  grew 
impatient  with  that  effort  and  ultimately  considered  the  various  maintainability  and  life-cycle 
costs  on  a  more  global  level. 

d.  Measuring  Success:  They  did  not  feel  certain  what  the  specific  IPPD  outcomes  or 
milestones  were.  Their  philosophy  was  to  be  aware  of  IPPD  processes  and  do  what  you  can, 
when  you  can,  while  concentrating  on  the  main  criteria  of  meeting  project  milestones  and 
keeping  customers  involved  and  happy.  In  this  project  the  use  of  IPPD  techniques  was  not 
necessarily  making  the  customers  happy. 

*  e.  Observations:  It  seems  this  project,  because  of  early  involvement  of  consultants, 

recognition  of  need  to  employ  IPPD,  and  contractors  acceptance  of  IPPD  requirements  had  a 
good  start  for  IPPD.  Use  of  IPT  sessions  involving  potential  users  enabled  managers  to  refine 
the  requirements  and  develop  an  HOQ.  However,  the  conflicting  demands  of  users  and  the 
users’  reluctance  to  patiently  gather  data  and  employ  IPPD  decision-making  methods  prevented 
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the  optimum  use  of  IPPD  value  analysis  tools.  IPPD  training  for  user  members  of  IPT  and  better  • 

skill  among  IPT  members  who  had  been  trained  (or  more  mentoring  during  IPT  sessions)  might 
have  resulted  in  overcoming  the  resistance.  The  interviewees  clearly  indicated  the  mentoring 
support  was  essential  until  program  personnel  had  enough  experience  with  IPPD  processes  to 
facilitate  their  implementation  at  the  IPT  sessions. 

The  PM  suggested  the  emphasis  on  increased  communication  with  and  involvement  of 
customers  on  the  IPT  was  a  facet  of  IPPD  that  was  very  beneficial.  It  helped  in  requirements 
determination  and  in  broadening  the  perspective  of  the  contractors  and  other  project  planners. 

Although  the  interviewees  didn’t  point  to  specific  examples  of  its  use,  the  material  covered 
in  the  DOE  course  was  also  reported  to  be  highly  regarded. 

NGT  (Next  Generation  Transparency)  This  project  is  developing  a  process  for  creating  a 
frameless  aircraft  transparency  (canopy/cockpit  cover)  by  an  injection  molding  process  (versus 
the  more  expensive/heavier  framed  transparency  apparatus  on  most  aircraft). 

a.  Interviewees:  The  researcher  met  originally  in  August  with  Bob  McCarty,  the  Program 
Manager,  and  three  members  of  his  team:  Mike  Waddell  (Deputy  PM),  Mike  Gran  (Chief 
Engineer),  and  Russ  Urzi  (through-the-canopy  technical  specialist).  Mike  Gran  had  not  been 
through  the  DSSM  course  but  had  taken  the  DOE  course  (and  eventually  the  IPPD  course).  He 
felt  he  had  enough  experience  with  IPPD  and  with  the  program  manager  that  he  could  provide 
meaningful  input  regarding  implementation.  Sixteen  other  members  of  the  IPT  took  the  DSSM 
course  in  May  1996,  but  were  not  available  (because  most  were  contractors)  for  interviews  on 
that  visit.  They  were  sent  Follow-up  Questionnaires.  Additional  follow-up  telephone  interviews 
with  Bob  McCarty  in  December  1997  and  April  1998  provided  the  updated  status  of  the  IPPD 
implementation  in  their  project. 

b-  Appropriateness  for  IPPD:  Although  team  members  took  the  DSSM  training  in  May 
1996,  the  project  funding  was  on  hold  until  June  1997.  Thus  at  the  first  interview,  the  project 
was  just  beginning  and  had  the  opportunity  to  demonstrate  use  of  the  IPPD  process  over  the  next 
two  years.  At  that  time,  and  at  a  later  interview  in  December,  McCarty  felt  this  project  had  a 
fairly  high  likelihood  of  positive  results  because  of  several  factors: 

-  Their  branch  (AFRL/V AVE-formerly  WL/FIV)  has  employed  similar  processes  for  years  in 
their  other  programs,  determining  their  customers’  requirements  and  doing  cost/risk/performance 
tradeoffs. 

-  Technical  proposals  and  the  SOW  under  PRDA  agreements  explicitly  state  that 
subcontractors  will  embrace  the  AFRL  model  process  desired  by  the  government. 

-  Two  years  prior  to  committing  of  funds,  system  customers  were  already  using  these  methods 
to  assess  risks  (industry  teams  supporting  JSF  program  and  F-15  SPO  were  assessing  risks  for 
transitioning  technology  and  considering  trade-offs  of  cost,  performance,  and  risks). 

c-  IPPD  Organization/Activities:  The  PM  indicates  they  have  three  IPT  subteams,  with 
both  government  and  contractors  on  each.  He  and  Mike  Gran  and  the  Boeing  PM  are  the  primary 
monitors  of  IPPD  implementation,  ensuring  appropriate  inputs  from  technical  specialists  such  as 
Russ  Urzi.  Communication  has  been  frequent,  with  weekly  meetings  and  teleconferences  to 
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discuss  problems  and  report  the  status  of  each  team,  and  with  an  active  use  Net  meeting  software 
to  facilitate  review  of  data  and  documents.  They  are  transitioning  to  a  secure  web  site  and  will 
*  be  using  web  site  databases,  a  process  that  Boeing  has  used  successfully  on  other  projects.  With 

these  methods,  Boeing  is  able  to  lead  a  team  of  five  scattered  subcontractors  and  keep  them 
enthused  about  the  process. 

In  addition  to  original  DSSM  training  in  May  1996,  the  PM  and  six  key  IPT  members  took 
the  IPPD  training  in  September  or  October  1997  and  six  other  members  took  either  the  DOE  or 
Design  to  Cost  training  in  1997. 

The  project  has  extremely  good  interaction  with  and  support  from  users/customers,  including 
both  Air  Force  commands  and  SPOs  and  potential  civilian  contractors  who  would  employ  the 
technology  in  automotive  windshields.  This  cooperation  is  working  well  in  these  early  stages  as 
the  customers  helped  the  team  identify  areas  of  technical  risk  and  define  system  requirements. 
They  even  reached  a  decision  with  the  F-22  community  to  use  the  F-22  vehicle  for 
demonstration  purposes  and  are  designing  a  dual-purpose  tool  for  molding  canopies  for  both 
F-22  drop-in  and  for  JSF  frameless  technology  maturation.  Most  of  their  activity  has  been  part 
of  the  development  of  the  HOQ.  Although  these  initial  definitions  are  subject  to  change,  the 
technical  or  engineering  requirements  as  well  as  the  cost  and  producibility  requirements  have 
also  been  defined,  and  the  initial  exit  criteria  have  been  established.  In  addition,  the  team  has 
worked  on  the  identification  of  technology  voids  (the  “roof  of  the  House”). 

Although  the  IPPD  process  outlines  six  steps  and  the  IPT  has  primarily  worked  on  the 
definition  of  requirements  phase,  the  PM  suggests  there  are  other  IPPD  considerations  from 
different  phases  that  are  addressed  in  parallel  rather  than  sequentially.  For  example,  they’ve 
already  been  employing  DOE  processes  to  determine  design  criteria  that  will  solve  the  crazing 
tendency  under  vaiying  temperature,  humidity,  and  load  factors.  They  also  have  been  thinking 
about  Value  Scorecard  issues  and  identified  key  large  investment  items  that  will  be  potential 
show  stoppers  on  the  Scorecard  at  a  later  time.  They’re  moving  more  fully  into  design  analysis, 
using  simulations  to  look  at  aspects  such  as  the  producibility  of  a  canopy  for  F-22,  optics,  and 
bird-strike  effects.  They  will  be  using  technology  worksheets  for  performance,  supportability, 
and  cost  to  document  the  results  of  their  analyses  and  will  roll  up  the  data  into  a  value  scorecard. 

d.  Measuring  success:  Although  the  IPPD  philosophy  is  evident  in  all  the  IPT’s  documents 
and  briefings,  and  they  plan  to  provide  extensive  training,  they  feel  the  real  measure  of  success 
remains  whether  teams  can  demonstrate  they  based  decisions  on  data  gathered  and  analyzed 

e.  Observations:  The  team  members  seem  to  have  bought  into  the  general  philosophy  of 
IPPD,  but  application  of  procedures  in  this  project  is  still  an  experiment.  So  far  it  seems  to  be 
going  well,  and  its  continued  success  may  be  affected  by  a  number  of  factors 

-  The  degree  that  vendors  can  become  fully  knowledgeable  of  the  IPPD  processes  and  their 
ability  to  collect  data  required  to  demonstrate  use  of  those  procedures  in  decision  making. 
Boeing  (a  prime  contractor)  has  employed  these  methods  in  their  777  project,  but  many  other 
«-  subcontractors  are  less  experienced  in  the  procedures  needed  to  gather  data  and  to  measure 

factors  needed  for  tradeoff  decisions.  A  key  seems  to  be  whether  government  team  members  and 
prime  contractors  can  hold  subs’  accountable  and  insist  they  (and  perhaps  help  them  learn  how 
to)  employ  the  data  collection  and  statistical  analysis  procedures  required  to  demonstrate  IPPD 
was  used  in  decision  making 
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-  The  ability  of  the  IPT  members  to  obtain  assistance  and  tools  to  help  them  in  this  effort.  The 
PM  indicates  that  James  Gregory  consultants  have  been  key  players,  providing  the  IPT  with 
decision  making  tools  and  an  electronic  version  of  the  House  of  Quality  and  facilitating  at 
meetings.  Additional  mentoring  will  be  needed  as  they  continue  through  some  of  the  other  steps 
and  decisions. 

OTH  (Over  The  Horizon  Radar).  This  project  is  wrapping  up  an  1 1-year  effort  to  improve  the 
ability  to  detect  small  low  flying  aircraft  at  great  distances.  They  are  working  in  conjunction 
with  (for)  the  Navy. 

a.  Interviewees:  The  researcher  met  with  Mitch  Beyer,  Doug  Cameron,  Lorraine  Flanders, 
Don  Harvey,  and  Greg  Merrill  in  one  group  to  discuss  their  efforts  towards  implementing  IPPD. 
Additional  discussions  were  initiated  with  the  PM  and  Ms  Flanders  to  review  the  status. 

b.  Appropriateness  for  IPPD:  This  project  is  11  years  old  and  in  the  last  year  of 
development.  From  the  initial  training,  the  group  indicated  it  was  too  late  to  implement  IPPD  in 
their  project.  Almost  all  of  the  requirements  decisions  and  design  specifications  had  already 
been  made.  Additionally,  this  project  was  developing  software,  and  the  group  could  not  figure 
out  how  to  adapt  the  IPPD  procedures  to  that  process. 

c-  IPPD  Organization/Activities:  There  has  been  no  continuing  formal  plan  for  IPPD 
implementation.  The  group  met  with  the  training  consultants  once,  immediately  after  the  initial 
IPPD  training,  and  they  determined  that  IPPD  technologies  would  not  work  on  this  type  of 
software  development,  especially  at  such  a  late  stage  in  the  project.  They  do  feel,  however,  that 
some  IPPD  techniques  are  valuable.  They  sent  several  of  their  members  to  additional  IPPD- 
related  courses:  six  to  the  DOE  course,  three  to  the  Design  to  Cost  session,  and  one  to  the  DSSM 
course.  For  ten  months  there  was  little  application  of  IPPD  procedures.  However,  as  the  final 
report  was  being  completed,  the  training  consultants  met  with  OTH  representatives  to  try  to 
develop  affordability  metrics  to  assess  the  relatively  new  Relocatable  Over-the-Horizon  Radar 
(ROTHR)  technologies.  They  are  exploring  the  potential  of  using  desirability  functions  to 
characterize  ROTHR  metrics  and  provide  a  way  to  evaluate  technologies  with  respect  to  a 
Customer  Satisfaction  Index  (CSI). 

d-  Measuring  Success:  No  efforts  were  made  in  this  area  since  they  did  not  utilize  IPPD 
techniques. 

e.  Observations:  The  group  liked  IPPD  and  saw  it  as  valuable.  They  just  could  not  realize 
how  to  apply  it  to  their  project.  They  indicated  the  Navy  also  uses  IPPD,  so  the  group  realizes 
there  are  some  potential  benefits;  but  due  to  the  maturity  of  the  research  project,  and  the  fact  they 
were  developing  software,  they  determined  it  could  not  be  applied.  Only  after  additional 
consultation  with  the  trainers  and  the  development  of  additional  metrics  with  other  pilots  are  the 
OTH  reps  reconsidering  the  analysis  of  technologies  in  a  manner  more  in  line  with  the  IPPD 
process.  This  is  a  good  example  of  how  the  implementation  of  IPPD  in  the  S&T  environment  is 
an  ongoing  learning  process. 
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VTVIDS  (Virtual  Interactive  Intelligent  Tutoring  System  Development  Shell):  This  IPT 
provides  an  opportunity  to  view  the  use  of  IPPD  processes  in  a  software  design  project.  The 
intelligent  tutoring  (IT)  shell  is  a  technology  that  was  initially  developed  in  1992/93  as  part  of  a 
Simulation-Based  Adaptive  Training  program  to  produce  an  authoring  tool  (software  product) 
for  embedding  instruction  in  software  simulations.  The  IT  shell  has  been  adapted  for  use  in 
several  training  programs  for  various  major  commands.  The  laboratory  also  uses  the  technology 
in  some  of  its  own  training  programs.  This  VIVIDS  IPPD  project  was  initiated  in  March  1997 
and  involves  both  upgrading  the  authoring  shell  capabilities  and  applying  the  VIVIDS 
technology  to  Air  Force  Special  Operations  Command  (AFSOC)  and  Air  Education  and  Training 
Command  (AETC)  training  projects. 

a.  Interviewees:  The  researcher  met  in  a  small  group  session  at  Brooks  AFB  with  five  of 
the  nine  IPT  members  who  had  taken  the  IPPD  training  at  Texas  Instruments  in  September  1997. 
These  participants  were  Jim  Fleming  (PM),  Winston  Bennett,  Terry  Jackson,  and  Craig  Hall 
(government  members  of  the  IPT),  and  Carol  Horwitz  representing  Coherent  Technologies,  Inc. 

b.  Appropriateness  for  IPPD:  As  a  typical  reiterative  software  design  process,  this  project 
is  important  because  it  can  help  determine  the  potential  for  using  IPPD  processes  in  a  software 
research  project.  There  is  a  definite  customer,  and  the  project  was  identified  as  a  pilot,  and 
initial  IPPD  planning  began  in  May  1997,  very  soon  after  the  project  kicked  off.  However,  the 
IPPD  training  did  not  take  place  until  September  1997,  after  a  number  of  decisions  regarding 
functional  specifications  had  already  been  made. 

c.  IPPD  Organization/Activities:  The  key  groups  in  the  IPT  seem  to  be  (a)  the  AFRL 
research  members  who  interact  with  potential  customers  to  determine  applicability  of  the 
VIVIDS  tool  to  their  training  course  needs,  help  outline  user  requirements,  and  test  and  evaluate 
the  tool  in  the  field;  (b)  the  CTI  contractor  who  develops  the  functional  specifications  for  the 
software  changes  that  must  be  made;  and  (c)  the  contractors  at  University  of  Southern 
California’s  Behavioral  Technology  Lab  who  actually  make  the  software  changes/enhancements. 
Because  the  CTI  representative  is  on  site  at  Brooks  AFB,  there  is  strong  team  coordination  at  the 
front  end.  Representatives  from  all  three  organizations  attended  the  IPPD  training  session.  In 
the  first  six  months,  there  has  been  little  post-training  interaction  with  the  James  Gregory 
consultants.  However,  the  IPT  members  and  consultants  are  attempting  to  meet  to  discuss  the 
ways  in  which  the  IPPD  metrics  can  be  translated  to  a  software  design  process. 

Because  of  the  timing  of  the  IPPD  training,  some  initial  actions  were  completed  before  the  IPT 
members  were  knowledgeable  about  the  IPPD  processes.  Major  requirements  had  already  been 
determined,  initial  decisions  regarding  functional  specifications  had  been  made,  and  two 
prototypes  had  already  been  determined.  Customer  requirements  were  outlined  at  TPIPTs  and 
program  review  sessions.  Although  a  House  of  Quality  was  not  developed,  the  IPT  members 
report  using  a  Quality  Function  Deployment  (QFD)  process  to  outline,  quantify,  and  weight 
(prioritize)  functionality  requirements. 

Team  members  indicated  they  had  been  dealing  with  cost-risk  analyses  for  years  and  that  they 
had  internalized  the  information  from  the  IPPD  process  and  could  use  that  knowledge  in 
informal  decision  processes.  In  development  of  the  two  prototype  approaches,  technology 
worksheets  were  not  used,  but  some  formal  spreadsheeting  of  the  functional  requirements  vs  cost 
factors  involved  in  many  decisions  was  provided.  These  results  are  summarized  in  the 
documentation  of  the  factors  considered  in  these  decisions.  Team  members  suggested  that  the 


customers  don’t  always  want  to  go  through  the  detailed  cost,  risk,  and  requirements  data  at  the 
beginning  of  the  project  and  that  the  IPPD  procedures  might  be  more  applicable  during  the 
evaluation  of  the  prototypes. 

d.  Measuring  Success:  Although  the  concepts  underlying  the  IPPD  process  (e.g.  cost,  risk, 
and  performance  tradeoffs)  are  discussed  at  review  sessions,  there  has  been  no  formal  process  for 
measuring  IPPD  implementation  progress. 

e.  Observations:  The  fact  that  IPT  members  were  not  trained  on  IPPD  until  after  a  number 
of  decisions  had  already  been  made  had  an  obvious  effect  on  implementation  of  early  IPPD 
procedures  such  as  the  HOQ.  There  is  a  great  degree  of  uncertainty  about  the  appropriateness  of 
applying  the  formal  IPPD  procedures  in  the  course  to  this  software  engineering  project.  At  this 
point  it  is  seen  as  a  less  efficient,  time-consuming  process  for  the  type  of  reiterative  design 
enhancement  efforts  that  are  common  to  the  software  domain.  This  may  be  a  case  where 
additional  collaboration  with  the  James  Gregory  consultants  would  enable  the  IPT  to  learn  how 
to  adapt  that  process  to  their  software  projects.  Or  it  may  be  that  the  consultants  and  IPT 
members  will  find  the  IPPD  process  is  less  useful  for  this  type  of  S&T  research.  At  this  point 
(six  months  after  initial  training),  the  quandary  is  not  resolved,  but  it  is  certainly  worth  the  effort 
if  the  IPT  can  afford  the  time  to  help  address  the  issue. 

WINDS  (Ballistic  Winds  -  Gunship,  Airdrop,  Bombdrop):  This  project  is  to  develop  a  lidar 
system  that  can  measure  wind  profiles  from  an  airborne  platform  to  improve  the  operational 
effectiveness  in  gunship,  airdrop,  and  bombdrop  accuracy.  Originally  a  prototype  was 
successfully  tested  for  gunship  customers  in  May  1996,  but  that  program  was  canceled,  and  the 
researchers  were  committed  to  demonstrating  that  the  technology  is  applicable  to  airdrop  and 
bombdrop,  as  well  as  commercial,  purposes.  At  the  time  of  the  November  interviews,  there  were 
no  immediate  customers  committed.  By  April  1998,  the  bombdrop  portion  was  on  hold  and  only 
the  airdrop  program  was  active 

a.  Interviewees:  Three  key  government  members  of  the  15  original  IPPD  trainees  still 
with  the  program  were  interviewed  (five  members  of  the  original  gunship  team  at  Robins  AFB 
are  no  longer  working  the  project,  and  three  contractors  from  CTI  were  not  available).  The 
members  were  John  Carr  (Airdrop  PM),  Everett  Wood  (Bombdrop  PM),  and  James  Root 
(Technical  Lead  for  third  generation  transceiver  development) 

b-  Appropriateness  for  IPPD:  Prior  to  cancellation,  the  gunship  effort  seemed  to  have  the 
components  for  effective  IPPD  decision  making  (technology,  mission  performance,  and  cost 
tradeoffs).  Although  the  lack  of  identified  customers  has  hindered  the  identification  of  specific 
requirements  for  air-  and  bombdrop  efforts,  questions  associated  with  development  of  a  third- 
generation  transceiver  for  wind  profiling  and  with  the  need  to  reduce  technology  transition  costs 
seem  to  be  appropriate  for  the  IPPD  process. 

c-  IPPD  Organization/Activities:  The  contractor  (CTI)  is  building  the  lidar  system,  and 
AFRL/SN  team  members  are  arranging  the  support  components  and  testing  on  the  airborne 
platform.  Sverdrup  is  contracted  to  determine  how  to  take  the  new  WINDS  data  and  apply  it  to 
the  existing  Computed  Air  Release  Point  drop  algorithm;  Raytheon  did  the  C-130  fuel  pod 
modification.  Most  communication  among  these  members  has  been  through  telephone  and 
electronic  means.  The  team  has  received  its  third-generation  equipment  and  will  be  conducting 
environmental  and  other  tests  in  the  lab.  The  goal  was  to  demonstrate  airdrop  usability  in  a 


March  1998  demonstration.  However,  technical  delays  have  pushed  that  date  to  September 
1998.  Initial  DSSM  training  was  provided  for  AFRL  air-  and  bombdrop  team  members  in 
November  1996,  and  new  IPPD  training  was  attended  by  three  CTI  contractors  in  April  1997. 
James  Gregoty  consultant  support  was  provided  to  CTI  in  May  1997  in  conducting  a  designed 
experiment. 

Team  members  have  met  frequently  with  potential  customers  of  the  airdrop  system,  and  the 
bombdrop  members  had  met  with  an  ACC  liaison  to  show  potential  utility  and  to  elicit  customer 
needs  and  more  specific  requirements.  The  airdrop  team  has  outlined  general  operational, 
technical,  and  affordability  objectives  and  exit  criteria.  A  technology  transition  plan  has  been 
built  and  coordinated  with  potential  users,  and  MANTECH  has  been  involved  and  has 
contributed  funding  to  take  the  current  third-generation  configuration  and  improve  aspects  for 
manufacture.  However,  the  team  determined  that  IPPD  process  documentation,  such  as 
developing  an  HOQ,  technology  worksheets,  and  value  scorecards,  would  be  too  demanding  on 
limited  time  and  resources  for  little  potential  benefit.  Therefore,  affordability  and  cost  decisions 
were  made  intuitively  rather  than  with  those  IPPD  tools. 

d.  Measuring  Success:  The  team  has  a  prepared  response  to  the  “Affordability  Pilot 
Program  Scorecard,”  which  shows  they  have  an  understanding  of  the  major  criteria  for 
measuring  success  of  an  IPT  program.  The  impression  is  that  they  are  fulfilling  some  of  the 
requirements  outlined  on  the  scorecard  (e.g.,  some  training  has  been  provided,  a  transition  plan 
has  been  developed,  affordability/produceability  issues  have  been  identified). 

e.  Observations:  It  is  clear  that  this  IPT  seems  to  embody  the  spirit  of  the  affordability 
initiatives  in  its  attempts  to  identify  performance/cost  tradeoffs  that  would  make  the  system  more 
attractive  for  military  customers  and  enhance  manufacturability  for  potential  transition  to  civilian 
customers.  There  is  a  sincere  belief,  however,  that  the  TDYs,  manhours,  and  extra 
documentation  and  briefings  associated  with  IPPD  implementation  are  added  costs  and  resource 
demands  that  are  not  reimbursed.  They  are  not  convinced  that  the  hard  part  of  the  IPPD  process 
implementation  (data  gathering  and  analysis  and  documentation  for  decision  making)  provides 
added  value  to  make  up  for  those  additional  costs. 

CTI  members  reported  they  felt  the  DOE  session  was  very  worthwhile  and  was  more 
beneficial  than  the  IPPD  session.  John  Carr  suggested  the  DOE  methods  could  be  used  in  the 
experimental  demonstration. 


